Request for Assistance: R Graphing Script for my QS project

Here is the R source for my QS weight graph. I get the data in Perl, run it through Template Toolkit, and run it out through here.

It is my code. I like my code. But I know it could be better, even without doing the database query in the R, which I believe it can do.

Here are the things I want to be able to do:

• I should have dates corresponding to the points. I have the data. I should put them in.
• Because I take measurements in the morning when I get myself together, I do not always remember to weigh or upload my weight. I want the graph to show this.
• I also know how many data points are averaged to make the red line. I want to overlay that over the graph, to show the quality of each rolling average point.

Suggestions and patches are accepted. I should say I want to do this with ggplot2 with Cairo as the image library, so I don't have to rely on X11 for rendering and can crontab this.

What's Up With All That Quantified Self Stuff, Anyway?

The way I normally present myself is as a large land mammal. Part of that is my height: at about 6'4", I'm well into the tallest %1 of the population. But that metric is not the crucial one, and as an adult, any attempt to hack my height would require surgery and be painful.

The crucial one is that I am, as of about 8 this 9:14am EST and by the measure of my scale, 265 lbs. This is not the worst I've been; through my second degree and my first IT job, I was up to 330-350 lbs. I don't recall exactly how heavy I was at worst. Understandably, I wasn't too encouraged to keep track, because those are not numbers to be proud of. 265 is a number to be proud of only in context of being some 50 lbs away from it, but I am somewhat proud of it. 265 lbs means I can shop in the Men's section and not the Big-and-Tall section for jeans. I was ecstatically happy when I got jeans that had a waist size of less than 40".

You might say "50ish pounds is pretty impressive. How did you do it?"

I honestly don't have an answer.

I have some ideas. As mentioned, my top weight came with my time in school. During this time, I was running myself ragged, where I would go to campus about 8am and come home and break fast somewhere between 10pm and midnight, then crash out and start again. Many times, I would go to company presentations where students traded pizza for resumes with recruiters, and that would serve as my food for a day or two. Eventually, it got to the point where today, I eat breakfast when I get to my desk and usually have several microwave lunches in the office freezer. A large contributing factor for my weight loss is that I eat more meals.

During this time, there were a few times when I joined a health club and tried to keep up. There's the promises they give, like "You'll feel healthier, you'll have more energy". I can't say that any of that ever happened. It was more bored and sweaty. "You'll be stronger" is also a promise, but you know, I honestly don't want that. I have broken a number of things unintentionally, putting my strength and heft at things that couldn't take it, and I was embarrassed each time. There is not a single thing I have to do, either personally or professionally, that taxes the strength I have. My relationship to my body is a bit like Communism and the State, in that I had hoped that it would wither away, but instead it grew and became a problem.

I'm happy to be down to 265 lbs. but I'm not satisfied. That just means I'm down from "morbidly obese" to "obese" on the health graphs, that I can buy clothes at Wal-Mart. (For all the obesity jokes about Wal-Mart customers, their clothes selection certainly doesn't cater to that demographic.) I've plateaued at 265 for a while, I think, but it might be that weight loss is still ongoing but my method (eating slightly better but more regularly, drinking water, adding slight more walking into my life) is just slow. I just don't have data, and you can't accurately judge effectiveness without data. I'm curious to get more, and I've put a FitBit onto my Christmas wish-list. Once I get a baseline, I can start adding activities or changing my diet, and judging them with some objective measure of effectiveness, rather than what I feel.

Which is why I'm looking into this, and want to learn more and do more. I want to track and analyze the habitat of this large mammal.

More with the Quantified Self stuff

This is me. Specifically, my weight over the last 20 or so days. That's the red line.

The blue line is the average of the weights reported over the seven days previous. There's only one weight recorded at first, so the starting points are the same. I'm told this is an excellent way to normalize the data, to even out the peaks and show the trend. It seems to work.

I was asked "Have you lost weight?" today, but conversation changed before I was able to explain that I can quantify the weight loss, and it's four pounds. That's probably a good thing.

Once I have some things together, I'll definitely put it up on Github. I'm even considering the next step, doing something -- gasp! -- commercial with it! The commerical version will be different, because I've coded this for one person, and I'll have to rejigger it for many parallel users.

• It's in MySQL, and I'm happy with it. I've been told by someone in the Perl community that I should really go with PostgreSQL, which I don't have a problem with, either. 
• I've been able to put all the logic in the query and just get the numbers, which I grab using Perl.
• Using Template Toolkit, I take the numbers and generate an R script. I'm not particularly happy with the means of running it, a system call in the Perl, but I'm happy that it runs, and I'm happy that I can mess with the view, more or less,without having to worry about the model. Metaphorically speaking, as this really isn't MVC. And you can write R that looks much better than this.

Why I'm Gobsmacked By Chrome

Yesterday, I put together a rant against Chrome's failure to handle tabs in Javascript. I was going to spend some time this morning developing a test I could use to demonstrate the problem, but first I had to get some work done.

While doing it, I had reason to use the application where I noticed the problem. Of course, I had to get everything going, because my machine went down for the usual Tuesday updates. And I went through the process and clicked OK before I realized that I was doing this in Chrome not Firefox.

And it worked.

For hackers, it is frustrating to have something fail to work when you don't understand why, but it is also frustrating for something to begin to work when you don't understand why. Chrome is on the update fast-track. Did they find my blog post and fix it overnight? Was my machine well overdue for a reboot, leaving my code encrusted in bit rot that needed a complete restart to clear out? Was it sunspots? Was it a hex? Was I just doing something stupid before? If so, what? I have no way to know why the problem is fixed, so I have no way of knowing when and if it will come back.

The fact that my code suddenly worked will probably be keeping my up nights for a while.

Granted, it is better for things to suddenly work for unknown reasons than for things to suddenly not work for unknown reasons, but both are a problem.

Why I Hate Chrome

Pretend you have a spreadsheet, for example an Excel spreadsheet. You want to do something with it online, because that's what everyone wants to do with stuff.

What's the simplest way? The simplest way is to make a TEXTAREA, and paste it in. If you don't want to make all sorts of subpages for every thing you'd want to put into a TEXTAREA, you make a thing with Javascript and AJAX it up to the server. You only have to reload the page if it works, which is cool. AJAX is cool.

Excel is kinda cool, in a way, in that, if you copy several columns and rows of a spreadsheet and paste it into a non-spreadsheet area, you get newlines for the end of a row and tabs to delimit columns. This is useful, because then you know that, when you have to turn that chunk of spreadsheet into a data structure, you can split on newline to get every row, and then you can split on tab to get every column.

That splitting is something I usually prefer to do on the server side. I write my server-side in Perl, and it can very easily split on newlines and tabs. I write my client-side with the help of jQuery, which I do not believe to be at fault.

I wish to send it up using jQuery's post, sending a JSON object. I put the TEXTAREA's value as a value in an object, and send it to the server. But, as early as I can tell in the process, Chrome's Javascript ignores the tabs, so if you had data that looked like AtabBtabCnewline1tab2tab3, it will pass it up as ABCnewline123.

I'm using Chrome 18.0.1025.56 beta-m right now. I also use Firefox. I mostly use Firefox for Facebook, because I can't load some Facebook games with Chrome. I don't normally mind that: it reminds me not to play around when I'm busy doing something else. Here, Firefox proved to me that Chrome is the problem, not my code. And I'm saved some, because my users for this tool use Firefox.

Still, it makes me mad.

Why I Hate Ruby

I'm taking the Software Engineering for SaaS course through Coursera, and that course is teaching Ruby.

Including the OOPier and LISPier parts. And I don't think like that.

Perl programmers should know $_. Consider $x = my_subroutine(). We see where the return value goes here, it goes to $x. Now, consider my_subroutine(). You know there's a return value. Now, where does it go? It goes into $_. Which means you can follow my_subroutine(); with print ; with no problem, because the first line drops it into $_ and the second line picks it up. New Perl programmers get infuriated by that magic until they understand it.

Ruby has a lot of magic. I understand very little of it.

This is making me cross. And it will make me cross until I start to understand the wizardry of it.

I Suppose I Have Been Trying QS Already

I suppose I should've said that the weight tracking is my first step toward Quantified Self that I've programmed, because I've used some apps written by others.

I have MyTracks and RunKeeper on my phone (not that I really ever use them), both of which can be used to keep track of what you do when you exercise. I have used MyTracks for a few walks, but the problem is more me and my lack of scheduled walks than anything with those tools. I can credit MyTracks with telling me that the walk from the bus stop to my building is 0.31 miles.

The coolest thing I've used so far is called Smart Alarm Clock. Key to this is the Android phone's capability to sense motion. If you tell it to wake up at 6am, it starts trying to sense you moving around, which you do more in REM sleep, which is not when you want to wake up. When it senses you being more still, that is when you are sleeping the lightest, which is the best time to wake you up. It really is like magic.

The weight tracker is somewhat inspired by the Hacker Diet, which I have not fully read. 

Dipping My Toe into Quantified Self

Quantified Self. "Self knowledge through self-tracking". And weight is what I want to track.

It was a moment where it just came together. "I can do that!"

I haven't hacked my scale to wirelessly tell my computer. That's a bit more than I want to do. Yet. (I've been looking at XBee stuff on adafruit, just thinking about that.) Right now, I want to collect data. What do I need for that? I need a way to save this data that I can handle in the AM before I really shower and wake up. Sound like a simple web page back-ended in a database to you? It did to me.

I roughed it out last night, and finished it up this morning. I did some gussying it up with CSS this evening after work, but that's incomplete. It looks much better, though.

I have only one true entry into the database so far, so graphing that information isn't a thing yet. I figure ten days of data is a good point to start taking a running average. I know how to automate that, but I don't have that data on that machine....

Maker vs Developer

I've noticed I have a tendency. I solve things once.

Let's take a specific example. Google Plus does not have RSS feeds. It does have an API, for which you can get a key, where you get a JSON output, and you can parse it down and push it into RSS. I didn't, at first, want to do my own stuff. There are a few people who tried to get into the game, but they fall down due to the limits put on G+ by Google.

So, I felt pushed to make it myself. In Perl, that's a little YAML to kick the API key out of the code so you can put it on Github, LWP to get the feed, JSON to parse it and XML::RSS to spit it into the desired format. I have it on a server somewhere. I want it specifically so I can put it into Google Reader, then plus the things I really care about, which ifttt then pushes out to my Twitter feed.

So, I'm the one who uses it.

Only me.

This could go on to Google+, Y U No RSS? but that isn't really the point here. The point here is that this, and similar tools I write, are designed to be things I want and end at being things I want. I think this is a very Maker thought. It's Adam and Jamie in their workshop, saying "I need a rig to drop a ball off a table, so how do I do that?" I have this problem. I solved this problem. I no longer have the problem. If you can do anything with it, fine. That's fine, but — and it pains me to say this — it isn't really scalable. Which is to say, it isn't a way to make scratching your own itch a business.

I do want to think more like an entrepreneur, trying to turn what can I do? more toward what can I do for others? which is how to make money off these things. This is why I liked the Greater Lafayette Startup Weekend. That's something that could be better for me.

How to manage several MySQL accounts

Here is something that I wish I had known a while ago.

So, I had my .my.cnf filled with the information for the login that was mine, which I used for test. Eventually, I started to work with a production database, and was given one login for general CRUD (generally for the web interface, but useful for day-to-day work as well) and one for creating and modifying tables.

I needed to be able to be in multiple ones at a time, which meant I couldn't just copy a config file with all the information about one setup to .my.cnf, then copy another when I needed that.

My initial take on the solution was to make a script and then an alias with all the correct login information (including the password), then call that when I wanted to get into the admin account. This puts the keys to the database into the process table, which is stupid and wrong and dangerous, but I didn't know a better way.

This morning, I thought to ask the DB StackExchange. Of course, I should've searched more thoroughly first. Here is what a sample .my.cnf would look like:

[clienttest]
host        = server.university.edu
user        = test
database    = test
password    = abc123

[clientprod]
host        = server.university.edu
user        = production
database    = production
password    = abc123

[clientadmin]
host        = localhost
user        = admin
database    = production
password    = abc123

[mysql]

prompt='mysql [\u][\d][\h]>'

You then start mysql by typing mysql --defaults-group-suffix=whatever, where the config says [clientwhatever].

That can, of course, be a [client] setting, so just typing mysql gets you into a default space. Adding aliases for easier typing is also useful.

alias test_db="mysql --defaults-group-suffix=test "
alias prod_db="mysql --defaults-group-suffix=prod "
alias admin_db="mysql --defaults-group-suffix=admin "

I think that prompt is too verbose, so mine is cut down to mysql [\u]>, but knowing which account you're in keeps you from accidentally killing something important.

And, of course, in the tradition of other pessimization removers I've blogged, it is good to have an alias to get into your .my.cnf.

alias mycnf="vi ~/.my.cnf"