February Check-In

I'm going to say that February was the worst month that I've had in a very long time. The main thing that happened is one of my favorite people in the entire world passed away. Short of something happening to my Mom, nothing could hurt as much. Along with my parents, for my entire life, my Aunt Lu was around. I will think of her on every Fourth of July (her favorite holiday) and Christmas Eve because we always went to her house. I often said she was hilariously funny, often unintentionally so. I loved going to visit her to hear about family members I never knew and what her life was like growing up. Since we knew she was sick, my siblings and I all made sure to visit her as often as we could during the month. She always loved to see us and I felt lucky to be there. If I could make people feel a quarter as good around me, as I felt around her, I'd be pretty happy. One of the last things she gave me was the dollhouse that someone made for her when she was a little girl. It has needed some work for a while and she gave it to me to fix it up. But only if I have fun doing it. If it's not fun, I shouldn't do it. I hope that I can get it into decent enough shape that I can donate it someplace where kids can play with it. I know she'd be happy if kids were playing with it.

I don't have too much to report on the rest of the month. I did drive more than usual. My odometer reads 35,572, so I drove 1,289 miles. I've been going to therapy for my knee and I think it's getting better. I need to get more flexible and I think that's just going to take time. So I have to be patient. I need to start watching what I eat better as I've been pretty lax about that. I'll try to do better in March. Lastly, if nothing else, I'm remembering that it's important to spend time with the people who mean the most to you.

10 years ago

February 25, 2016 was the 10th anniversary of my blog. Technically, it's the 10th anniversary of my using wordpress on the blog. I believe I registered coldandheartless in 2000, when my nephew was born, though I'm not entirely sure. But since I have pictures from when he was born posted, I'm pretty sure. And I have some pictures in directories named 2003. Yay for me in naming the directories correctly. So I know I had the website before 2006. But moving to wordpress looks to have happened in 2006. So let's celebrate that.

My first blog post tells what I thought the blog would be about, which was my house. That's still basically true as, ten years later, I am still working on my house. But it has also evolved to be a place for me to put notes on anything that I want to remember. This could be my new year's resolutions or how to write a bit of code. You could sort of say that it's my own personal google. And something that does cause me to chuckle is when I google something and it gives me a result on my own website.

So a belated happy anniversary to the blog part of coldandheartless. I'm happy that it's still around and still a bit bummed that I lost a bunch of posts between 2008 and 2010. (Lesson learned, don't run the server at your house. It's worth it to pay a cloud provider. FYI, I use linode.) Here's to 10 more years of me learning new stuff and documenting it here.

Producing a Lot of Crap

I was reading some notes a person took on a book and found something that I really like a lot. In the book "The Geography of Bliss" (which I haven't yet read, but now have a hold on at my library), the author writes that in Iceland, they produce a lot of crap. I guess the idea is that lots of people there produce art. So, of course, with so much art being produced, a lot of it is junk. But instead of thinking that's a bad thing, it turns out that all the crap produced encourages even more people to produce art. So basically, all the bad art is working as fertilizer to help make more art. And every so often, something great is produced. I think that's great! My immediate reaction is that I should start producing more and just accept that most of it will be bad. But, hopefully, it will work as fertilizer for me as well and maybe I'll make something great.

Uploading a File in a Rails Site to Use to Update

I want to upload a file to my rails webpage, but not store it in the database. I just want to upload it and then get the data out of it and update the database with the information in the uploaded file.

I have a settings controller that I use for most administrative tasks. So this is where I'm going to put the form for the file upload.

View:

<%= form_tag({action: :upload}, multipart: true) do %>
	Upload xml file of abstracts: <%= file_field_tag 'abstracts' %>
<%= submit_tag %> <% end %>

Routes:

 resources :settings, only: [:index, :edit, :update] do
    post 'upload', on: :collection
  end

Controller:

  def upload
    file_data = params[:abstracts].tempfile
    match_abstracts(file_data)
    flash[:notice] = "File uploaded"
    redirect_to settings_path
  end

Then the method match_abstracts is where I do all the work I want to do on the file. Right now, all I have is the information matching names in the uploaded file with names stored in the database.

  def match_abstracts(file_data)
    File.open(file_data) do |file|
      @doc = Nokogiri::XML(file)
      elems = @doc.xpath("//Speaker")
      @applicants = Applicant.without_abstract
      @applicants.each do |applicant|
        elems.each do |e|
          if e.parent.at('AcceptedTrack')
            if ((e.at('FirstName').text == applicant.firstname) && (e.at('FamilyName').text == applicant.lastname))
              logger.info "=="
              logger.info "TITLE: #{e.parent.at('Title').text}"
              logger.info "ABSTRACT: #{e.parent.at('Content').text}"
              logger.info "#{e.at('FirstName').text}"
              logger.info "#{e.at('FamilyName').text}"
            end 
          end
        end
      end
    end
  end

This just prints to the log the information that I'll need to use to update the database. Next I need to write another method, probably called update_applicant with the commands to add this information to the applicant's record.

Using Ruby on XML Files

I need to copy some data from one system to another. On the first system, I can download an xml file. The second system is a ruby on rails website. So I'd like to them upload the xml file to the site and have it enter in the data. But since I'm not that familiar with working with xml files, I thought I'd work on it from the command line first.

In ruby, there's a gem called nokogiri which works with xml (or html) files. Here's the sample xml file that I'm using:



 TEST CONFERENCE 2016
 
  5
  Diversity, diversity
  This is all about diversity.
  This is all about diversity.
  
   George
   Washington
   [email protected]
   WH
  
  
   George
   Washington
   [email protected]
   WH
  
  Abstract
  Diversity and Inclusion 
  Diversity and Inclusion 
 
 
  6
  Higgs, higgs, higgs
  I've got *something to say* out the **Higgs.**
  I've got *something to say* out the **Higgs.**
  
   Abe
   Lincoln
   [email protected]
   The White House
  
  
   Abe
   Lincoln
   [email protected]
   The White House
  
  None
  Higgs Physics
  Higgs Physics
 
 
  7
  Oh the Lovely Higgs
  Some interesting info about how pretty the Higgs is.
  Some interesting info about how pretty the Higgs is.
  
   Grover
   Cleveland
   [email protected]
   WH
  
  
   Grover
   Cleveland
   [email protected]
   WH
  
  None
  Higgs Physics
 

And here's the script I wrote:

#!/Users/maryh/Software/rubies/2.2.3/bin/ruby

require 'nokogiri'

File.open("Abstracts.xml") do |file|
	@doc2 = Nokogiri::XML(file)
	elems = @doc2.xpath("//Speaker")
	myfirst ='Abe'
	mylast = 'Lincoln'
	elems.each do |e|
		if (e.at('FirstName').text == myfirst) && (e.at('FamilyName').text == mylast)
			puts e.parent.at('Title').text
			puts e.parent.at('Content').text
			print "#{e.at('FirstName').text} " 
			print "#{e.at('FamilyName').text}\n"
		end
	end

end

First, I open the file and open it as an xml file with nokogiri. I then put in elems all the xml fields labelled "Speaker". I then cycle through elems looking for the string "Abe" and "Lincoln". If I find them both, I print out the field "Title" and "Content" which are both in the parent field from "Speaker" and then I print out the first and family names.

This is very basic and there are a lot more things that I need to do. I need to match the speaker info with names stored in a model in my rails app. And then update that entry with the title and content if I find a match.

Using Ruby on CSV Files

I'm working with some different systems for a conference I'm working on. We're using RegOnline to handle the registrations. One of the things that I need to do is get a list of people who have registered on RegOnline and post them on our conference website. Our conference site is plain html and I don't want to have to type in each person who registers. I can download a csv file from RegOnline. I want to write a script that reads the downloaded csv file and makes an html file for the website. I hadn't done much with csv files before, but here's a basic script that pretty much does what I want. (I still have to put html tags in, but that should be pretty easy.)

#!/Users/maryh/Software/rubies/2.2.3/bin/ruby
require 'csv'

unless ARGV.count == 1
	puts "SYNTAX:  generate_list_of_participants "
	puts ""
	puts "Expecting the csv file downloaded from regonline."
	exit(0)
end

inputfile = ARGV.first

my_csv = CSV.read("#{inputfile}", { headers: true })
# my_csv now has an array where each line in the file is an entry
# headers: true means the first line is headers and we don't want them
my_csv.each do |e|
	if e[4] == 'Other'
		puts "#{e[1]} #{e[2]} #{e[5]} #{e[6]}"
	else
		puts "#{e[1]} #{e[2]} #{e[4]} #{e[6]}"
	end
end

The first part of the script is just giving the syntax and checking that there's a filename on the command line. The second part, reads the entire file into and array (of arrays) called my_csv. The bit headers:true means that the first line in the csv file is a header row and it won't read that line in.

Then I just cycle through the array and print out the fields that I want. My csv file has a job in field e[4]. If it's other, I want to print what they typed in for their job (in field e[5]) instead of the 'Other' string.

That's pretty much it. There are some other things that I need to do, but to get started, this does the job.

Things to do:
-sort the array in alphabetical order on field e[2] (lastname)
-check that the attendee has actually paid, if they're submitting a check or wire transfer

Paperclip with Spoof File Checks

I have a rails site that accepts pdf files as uploads. Based on previous experience, I have found that trying to validate on content-type causes lots of problems. There are many browsers that send weird content types with file names. For pdf files, they should always be using 'application/pdf' and most browsers do. However, this morning I got email from someone who couldn't upload his file in one browser. He told me he used a different browser and it then worked. I searched the log files and found a bunch of error messages, but the main bit was this:

INFO -- : [paperclip] Content Type Spoof: Filename Tom_Smith_CV.pdf (x-unknown/pdf from Headers,
...
 content type discovered from file command: application/pdf. See documentation to allow this combination.

If I understand this correctly (and there's no guarantee of that), it's saying that the file is being uploaded with a content-type of x-unknown/pdf, but the file command (file --mime-type -b <file>) is returning application/pdf.

I believe that this mismatch is caused by the client browser being configured incorrectly. (And yes, it is. Check out this website.) Why? Because I was able to duplicate the problem by editing mimeTypes.rdf in my firefox profile directory. I simply changed anything that showed application/pdf to x-unknown/pdf and restarted the browser. I then got the same error.

This check is part of a new spoof detector in Paperclip version 4 and up. (I'm using 4.3.2.) Now based on lots of problems I've had before with validating content type, I put the do_not_validate_attachment_file_type :cv line in my model. Then I write a special validator that basically just checks the filename that it's a pdf file. This isn't ideal, but it has worked for me, until now.

This page explained that the spoof detector is not affected by the do_not_validate_attachment_file_type helper. That same page also gave me the workaround to turn off the spoof detector for now. I made a new initializer called paperclip.rb that contains:

require 'paperclip/media_type_spoof_detector'
module Paperclip
  class MediaTypeSpoofDetector
    def spoofed?
      false
    end
  end
end

This basically just returns a false value for anything checking against the spoof detector.

I realize that none of this is ideal. However, I really don't understand things enough to know how to set up paperclip to properly do file validations while also accepting all the files that we should accept.