How does one plot 5.2 million XY data points?
I ran into this while working on a paper submission. This is one thing one does not lack for when doing evolutionary computation: size of data sets.
Matlab seems to become dog slow and unstable when trying to plot large numbers of data points. The interface bogs down such that trying to re-label axes is a real chore.
I tried out the GNU R package, and had it crash on trying to read in the data set.
Then I started going through plotting packages in the FreeBSD ports system. That’s where I came across the GRI package. This is an open source, GPL licensed graph plotting language. It has simple examples available online. As its documentation notes, it is a package with a fairly shallow learning curve. Its interface is entirely command-line, and its output option is PostScript. In interaction, it simply outputs PostScript graphic after Postscript graphic, simply named “gri-nn.ps” in the current working directory. One can import data from an ASCII file, where columns are separated by white space.
So that’s exactly what I did. 5.2 million data points in, one 151MB Postscript graphic out. Ghostscript can convert that to PDF, which can then be converted to all sorts of raster-based graphics formats. It’s not a perfect solution, but it is a working solution.
Back to the grind for me…
Back in 2006, we passed on a computer system to a friend of ours who had none at home. Since then, she has gotten interested in digital photography and wants to produce some designs for CafePress, as well as the usual email, browsing, and writing. This is complicated by the fact that she and her husband don’t really care to spend the bucks for broadband.
Yesterday, she was shopping and ran across a sale on a Compaq Presario F750US laptop computer, at $450. She ended up buying it after consulting with Diane and I. She is planning on using some of the free WiFi hotspots in her vicinity, such as the local branch of the public library, in order to use the Internet.
We talked a bit later, and I mentioned that in keeping with the budget she’d like (spend as little as possible), that there were a number of open source or free applications and utilities that she should look into. I’ll append what I wrote to her. Please add other suggestions or disagreements on my suggestions to the comments.
I lived from late 2003 to early 2007 in Concord, California. The news story above concerns Pleasanton, a community near Concord. It seems to me to be much todo about nothing.
At basis, it concerns an apprentice falconer who simply wants to have an outside mews in her backyard. This, apparently, was not specifically mentioned in Pleasanton’s municipal regulations, though there are rules permitting the keeping of chickens and other “fowl”.
Now, this likely would never have attracted attention from anyone, but the apprentice falconer also happens to be the current mayor of Pleasanton, Jennifer Hosterman. That brought on various attempts by her political opponents to make things as difficult as possible for Hosterman. This latest report notes that Hosterman was given a temporary permit for her mews.
Some of the commentary reported comes straight out of clueless idiocy, such as the opponent who said of hawks at every opportunity that they were “trained to kill” and that the community would be at risk of Hosterman’s hawk attacking humans and pets. First off, no one needs to “train” a raptor to kill; that’s part of their inherited behavior. Raptors get trained to associate the falconer and good things happening, primarily the availability of food, and to hunt in particular ways. What they are likely to attack, though, primarily comes from untrained instinct. Birds taken from the wild, as it seems Hosterman’s hawk is, generally are quite reticent when it comes to people other than their falconer and unfamiliar animals, like domestic dogs and cats. (A domestic rabbit, though, would be very likely to trigger a response from a hungry hawk.) One suspects here that idiocy in pursuit of political ambition is simply par for the course.
Diane and I have made neighborhood or community regulations a deciding factor in places that we have lived to avoid situations such as Hosterman is facing. We purchased a home in the Sea Isle subdivision of Galveston, Texas in preference to another subdivision precisely because of vagueness in the neighborhood association rules in the other subdivision. In general, there do need to be some limitations on things people can do when packed in like sardines in subdivisions, but there should also be a recognition that those things that do not infringe on the peace of their neighbors or their property values are also not the business of their neighbors.
President Bush signed off on an exemption so that the US Navy can argue for continuing to use mid-frequency sonar systems in training exercises off the coast of southern California. While there is still a court injunction, the article notes that this exemption strengthens the Navy’s position in pushing for use of the mid-frequency sonar during exercises, and everyone will be back in court soon to hash this out.
The article quotes the following:
“The president’s action is an attack on the rule of law,” said Joel Reynolds, director of the council’s Marine Mammal Protection Project. “By exempting the Navy from basic safeguards under both federal and state law, the president is flouting the will of Congress, the decision of the California Coastal Commission and a ruling by the federal court.”
The Navy has long held that its compliance with various and sundry rules and regulations is voluntary. Mr. Reynolds should know by now that the Navy’s end move is to tell everybody to get stuffed, and do what they were planning to do anyway. The real trick here is to advance, so far as possible, the cause of minimizing damage to marine mammal stocks due to Navy actions, while not doing so in such a way as to cause the Navy to stop listening altogether to environmental activists or more moderate voices for conservation. The rhetoric in the above quote isn’t helpful in this regard. The idiom about one may as well be hung for a sheep as for a lamb applies here.
And the long-standing tactic in environmental activism of confusing and conflating the different systems of sonar that the Navy uses doesn’t help with the current situation, either. While the general public may be impressed, the people in the Navy tasked with dealing with these issues certainly aren’t put in a good position when research results about one type of sonar are falsely used to argue that the Navy should stop using either other types of sonar, or all sonar systems. There has been some improvement on this score over the years, but things could be better.
There are real problems to be solved concerning the responsible use of military sonar. Getting to the point where solutions are possible is going to be tricky; the Navy does have a legitimate purpose for use of these systems that has not gone away, and there are also legitimate concerns about balancing the Navy’s use of these systems with the risk that they pose for marine mammal stocks. The viewpoint that the Navy might be somehow barred from use simply isn’t feasible, and the possibility that the Navy might stop interacting with civilian interest groups of various sorts to find that appropriate point of balance would be unacceptable.
Hat tip to Ed Brayton.
Speaking of brazen antics, Glen Davidson over on the AtBC forum blew the whistle by posting the “Expelled Challenge” FAQ. Apparently, these folks are running scared that their project will be little more than one step up from “direct to video” projects, and are coordinating mass attendance of students and their parents from literalist-Christian schools. They are providing what amounts to a kickback to school administrators for movie ticket stubs from attendees who go to the “Expelled” movie during its first two weeks in the local theater.
Your school will be awarded a donation based upon the number of ticket stubs you turn in (see submission instructions in FAQ section). That structure is as follows:
0-99 ticket stubs submitted = $5 per ticket stub
100-299 ticket stubs submitted = $1,000 donated to your school
300-499 ticket stubs submitted = $2,500 donated to your school
500 ticket stubs submitted = $5,000 donated to your school
Each school across the nation will be competing for the top honor of submitting the most ticket stubs with that school having their $5,000 donation matched for a total donation of $10,000!
Notice that the ticket stubs become worth $10 each to participating schools at the maximum on their scale, a value that is likely higher than the actual ticket price. We knew before that antievolutionists fell into the “more money than sense” category, but this provides abundant confirmation that someone is very, very worried about reception of this film, and is willing to pay a premium to artificially drive up ticket sales. And note the specific recommendations given to schools for this kickback drive:
Q: What’s the best way to get our school families to come out to the movies?
A: In speaking with Christian Schools, we’ve found that hosting a school-wide “mandatory” field trip is the best way to maximize your school’s earning potential. Send a field trip home with your middle school and high school students, have each child pay for their own ticket, then collect the stubs at the door once you get to the movie theater. With this model, you also will be able to benefit from the ticket stubs purchased by parents who choose to come as well.
Wow… what an opportunity to take children away from classrooms, fill their heads with obnoxiously delivered misinformation, and profit off of it. I’ve said it before and I’ll say it again, exposure to antievolution decays morals.
Megan Downs at Florida Today called for people to send her their opinions about “intelligent design”. I fully expect the resulting article to be a mess, but I’d like to be pleasantly surprised. In any event, here’s what I’m sending along:
I’ve been interacting with “intelligent design” creationism (IDC) advocates since presenting at one of their conferences held in 1997. There, Phillip Johnson made clear that the theme was to “legitimate the question”, meaning that they were seeking a concession or acknowledgment that IDC was properly and rightly considered a scientific endeavor. I and other critics there asked a simple two-part question: what would an “intellligent design” hypothesis look like, and how would we test it? They had no answer then, and they have no answer now. This alone would disqualify IDC as rising to even a minimal level of science, but since 1997 we have found that IDC’s origin stemmed from a collaborative act of deceit.
Part of the Antievolution.org domain is an online web-based bulletin board system; let’s call that the AE Public BB. It has been operating since about 2002 with the current software, the Perl-based IkonBoard system. IkonBoard has pretty much become a legacy system without much in the way of support or user development, but the instance I have has had various modifications that provide useful features. Back when we were having issues with putting too much load on our old server, the IkonBoard installation was the application that showed the fewest problems, though even it was affected.
The point of having a BBS is to get a good community of commenters, and this the AE BB has done, in large part due to its being the spillover discussion area from the Panda’s Thumb weblog.
Early on, my idea was that threads on the BBS would bring together interesting and useful knowledge about topics, which could then be hosted as resources in themselves. I even made a start then of hand-saving the HTML of some threads to put in the main AE domain space. But at the time the BBS was using a “dbm” style database, and my only access to it lay through the IkonBoard interface. Since then, I’ve switched the database to MySQL and programmed a couple of accessory pages for mining information from it.
IkonBoard, like many other BBS systems, uses an arcane URL structure to specify threads on the BBS. It also insists on inserting a session identifier into its URLs generated dynamically. All of those things tend to make search engine spiders either give up entirely, or for the backends to give little weight to pages that do manage to get indexed. While I’ve added a hack that presents search-engine-friendly URLs to search engines, it still seems like pages don’t get indexed, or at least don’t reliably show up in searches.
So, given the early intentions and the more recently acquired means of doing something about it, I’m pleased to unveil the AntiEvolution.org Public Bulletin Board Archive, a page that presents the content of the AE Public BBS via links with meaningful names (they are taken directly from the topic titles). The procedure behind it will make it simple to update this archive periodically, and I’m intending that to be every month or two. There is a web-interface application in PHP that queries the MySQL database to find all topics that are cleared for public viewing. That application builds a page that has the HTML for a links page, and a set of commands for retrieving each topic as a single (sometimes very long) page from IkonBoard. Because IkonBoard generates each page, all the original links work, and the theme carries over. A user interacting with the pages will be shifted over to the IkonBoard BBS system and its dynamic, up-to-the-minute content.
So I think that this will serve the purpose of saving the sometimes useful, and more often entertaining, exchanges that take place on the BBS. There are definitely things there that should be remembered, and not be treated as evanescent and disposable interchange.
Barbara Forrest weighs in again on the faux neutrality of the Texas Education Agency and their mistreatment of Chris Comer, the director of science curricula they forced to resign in November. Don McLeroy gets some attention, too.
Pim van Meurs has an excellent post on Panda’s Thumb, Ignorance by Design, with loads of links to the various entities and people involved in the Florida antievolution dustup.
And please, get involved if you are in Florida. The new science standards need your support now.
Pamela Anderson thinks that donating a cut of the gate for “Blond and Blonder” to PETA, the animal rights extremist organization, will increase ticket sales for the movie. It’s your choice, of course, but my feet will be staying far away from the theaters on this one.
A nice uplifting article about a goshawk who was in a vehicle when the vehicle was stolen. The thief apparently dumped the goshawk out of its travel box. The falconer put out a request for information in the local newspaper with a reward, and while there were a lot of responses, one of them turned out to actually bring falconer and goshawk back together. The big issue was that the goshawk had its jesses tied together to a glove, and so any snag could immobilize the goshawk and prevent it from hunting for food. That put a time limit on how long the bird could survive after the theft if not found.
I recall a kestrel I had that made an escape with a yard-long leash on. Though I didn’t recover the kestrel, I’m pretty certain it was fine, since it knew how to divest itself of its leash and jesses in minutes on its own. As various people have noted since, when working with animals, it pays if the trainer is somewhat more intelligent than the animal; some of the raptors I’ve encountered apparently tip the scales in the other direction.
The count of county school boards in Florida adopting antievolution resolutions is now claimed to be at 12. And PZ Myers at Pharyngula notes that the various school boards have been adopting the same resolution.
That, PZ explains, means that there are people working stealthily behind the scenes to bring this about across the state of Florida.
When it comes down to this all going to court, as it looks more like doing all the time, that kind of widespread action involving a large number of people, some of them obviously challenged in the intellectual gifts department, is not going to remain a secret. It will come out… the question is, when? Will it be like the bombshell in the Kitzmiller case where Bill Buckingham and Alan Bonsell had to face up to their inconsistent testimony over the matter of who donated the money to buy 60 copies of “Of Pandas and People”, when the plaintiffs were able to produce a copy of the check Buckingham handed over to Bonsell for that very purpose? Somebody is going to give up the identity of the folks organizing this whole thing, they’ll be subpoenaed to obtain their correspondence on the topic, and I’m predicting that the result will be significant for a court looking into the purpose prong of the Lemon test.
There’s a website in Florida parroting the Discovery Institute’s faux concern for “academic freedom”, the Florida Coalition for Academic Freedom. So far, the “coalition” appears to mainly be the person of David Gornoski, Bartow High’s 2006 valedictorian. Gornoski’s evolution in antievolution is apparent, starting from his 2006 address at the Bartow High School graduation ceremony (he openly advocated teaching “intelligent design” creationism as science in public schools, and said of “anti-religious groups” to “Bring it on”), through a late 2006 effort to place a nativity scene on public grounds, and now to the DI-approved “academic freedom” for teaching “strengths and weaknesses” of evolution.
From Gornoski’s 2006 press release concerning his speech supporting IDC:
The student went on to address the hotly debated issue of Intelligent Design in public schools: The majority of our country believes in an intelligent designer with a plan for our lives yet public schools teach children that we are but accidents with no real purpose. I feel sorry for any student convinced through evolution science that mating and protecting territory is their highest purpose in life. It is no surprise then that since the decades when creation was replaced by survival of the fittest violent crime has surged over 330% according to Specialty Research Associates. Gornoski went on to provide a personal experience of the supposed violence increase. When you’sve seen students laugh as they stomp on a person’ss head, you’sve seen this statistic.
It seems obvious that being a valedictorian at Bartow High School does not necessarily indicate competence in logical argument. In the above, one finds the summary to be comprised of fallacies one after the other: argumentum ad populum, strawmen, and post hoc ergo propter hoc. (The repeated references to general audience approval indicates that this is a lacuna that affects more than just those at the top of the class.)
I wonder why obviously sincere, if somewhat over-zealous folks like David Gornoski never seem to tumble to the fact that they are participating in a con job. Gornoski’s original stance was a pleasantly honest, revealing advocacy of unconstitutional acts. In the nativity scene caper, Gornoski slid toward deceit with the statement that no, his nativity scene wasn’t about religion, but rather the historical relevance of Jesus. That sort of fibbing was just a warming-up period, though. This latest manifestation of Gornoski’s evolution follows the DI-approved tactics for getting the same old, tired, bogus antievolution arguments fed to students as if they were science, and a set of more mendacious tactics would be difficult to find. It includes the DI talking point about not mandating the teaching of “intelligent design” creationism. It parrots the DI-speak about wanting to teach “more” evolution, when what they mean is that they want equal time for evolution and antievolution. And an antievolution site without a quote-mine of Darwin would be incomplete, so obviously Gornoski’s site dishes it up in the header of his theme. Darwin’s message was that his book was too short to collect all of the opposing arguments, too, not that he would be seeking that fair result. In our modern situation, the clear message is that science classes have too little time to spend on teaching students both science and anti-science as if they were the same thing. Evolutionary science has passed muster through rigorous test and scrutiny; the “weaknesses” that the DI promotes are just the same old discredited tosh that has been seen from religiously motivated antievolutionists for decades and centuries. Evolutionary science is accountable through the record of hard work of scientists seen in the scientific literature, and antievolutionary drivel is not.
Instead of the forthright, honest, though misguided advocacy Gornoski started out with when he plumped for IDC back in 2006, the current website seeks to obscure those things that clearly speak to Gornoski’s obvious religious motivations in pursuing his goals. Probably the clearest sign of this lies in Gornoski’s misuse of the Darwin quote about achieving a fair result; if Gornoski really believed in his misinterpretation, one should find that his own list of external links would include both IDC and other antievolution advocacy, and sites explaining evolutionary science and criticism of antievolution. Instead, Gornoski’s blogroll is this:
* Dissent from Darwin
* Evolution News
* Explore Evolution
* Florida Science Standards Review
* Icons of Evolution
It only contains links to DI-sponsored websites and one link to the public commentary page for the new Florida science standards. I guess we can add “hypocrisy” to the moral decay that DI-style antievolution induces in its advocates.
The Florida Citizens for Science weblog picked up on a new report of what is now an old story, that the Taylor County School Board passed a resolution against the draft Florida science standards, back in November.
Upon motion by Danny Lundy, seconded by Darrell Whiddon the Board adopted/approved the: 1.) Resolution regarding the new Sunshine State Standards for Science.
The adoped resolution is as follows:
Whereas, the Florida Department of Education has drafted and is now proposing new Sunshine State Standards for Science, the Taylor County School Board opposes the implementation of the new standards as currently presented.
Whereas, the new Sunshine State Standards for Science no longer present evolution as theory but as “the fundamental concept underlying all of biology and is supported in multiple forms of scientific evidence,” we are requesting that the State Board of Education direct the Florida Department of Education to revise/edit the new Sunshine State Standards for Science so that evolution is presented as one of several theories as to how the universe was formed.
Whereas, the Taylor County School Board recognizes the importance of providing a thorough and comprehensive Science education to all the students in Taylor County and to all students in the state of Florida, it recognizes as even more important the need to present these standards through a fair and balanced approach, an approach that does not unfairly exclude other theories as to the creation of the universe.
NOW THEREFORE, BE IT RESOLVED by the Taylor County School Board of Taylor County, Perry, Florida, that the Board urges the State Board of Education to direct the Florida Department of Education to revise the new Sunshine State Standards for Science such that evolution is not presented as fact, but as one of several theories.
Here’s a problem: the Taylor County School Board, unanimously, doesn’t have a clue about what constitutes a scientific theory. Some telltale signs of a valid scientific theory:
- It generates testable hypotheses
- It appears in the scientific literature
- Tests of its hypotheses occur, support the theory, and are also published in the scientific literature
- Advocates of the theory respond substantively to criticism, also in the scientific literature
- The scientific community generally assents that the theory is well-tested
Cold fusion falls down at having tests support the theory and, of course, the general assent of the scientific community. “Intelligent design” creationism, and other forms of creationism, don’t score on any of these signs; they are not scientifically accountable, and responsible educators should not be wasting students’ time in science class with non-science. That, besides the obvious drawback that all the ‘alternatives’ in question would be various labels for the same ensemble of religiously motivated antievolution arguments.
In contrast, real scientific advances may go through periods (sometimes long periods) of critical scrutiny before general acceptance occurs. Things like transposable elements in genetics, punctuated equilibria in paleontology, the endosymbiotic theory, and the bacterial theory of ulcer causation all managed to become accepted because their advocates spent their time actually doing the scientific work, including serious and substantive responses to criticism, that convinced the scientific community of the worth of their ideas. It is because of that dedication to hard work that their ideas have achieved respect.
The advocates of religiously motivated antievolution, on the other hand, despise that sort of hard work. They think that subverting the political process is an easy and cheap way to circumvent the scientific process. It also neatly sidesteps any need to consider or even respect critics and their criticisms.
The Taylor County School Board has cast its lot with the layabouts and scalawags. The students in Taylor County will receive the message, loud and clear, that hard work is not to be respected in Taylor County.
The Palm Beach Post’s new weblog, “Extra Credit”, aims to cover issues in education, both K-12 and higher education. One of the current topics is the treatment of evolution — and absence of “alternatives” — in the new Florida science standards. Amidst various news stories telling of school board members and even Florida DoE employees taking up religiously motivated antievolution as a cause, the PBP blog has a story about two educators who support the new standards.
I found the story to be a bit on the cutesy side, but I did appreciate that the Florida media is picking up on reasonable people supporting the new standards.
I’m just back home from the holidays. This year, Diane and I went to Texas. While out-of-state hunters still get the shaft, at least they do offer a not-horrendously-priced 5-day small game license. Yes, getting multiple ones of that still worked out better than the full out-of-state license for a year. Anyway, if you’ve wondered if I fell off the face of the earth, no; but I could have hung out a “Gone to Texas” sign. Once I get some things organized, I’ll put up some pictures.
When we got home, we had to do a few fix-up things concerning the gate in the yard, the hawk mews, and the game bird pen. But for once the temperature was mild enough here to allow us to go out in T-shirt and jeans. That apparently is not expected to last long.
As with many guys in the USA on New Year’s, I’m in a room with a big screen TV and college football playing. The particular game is the Konica-Minolta Gator Bowl, but the thing that motivated me to write this entry is a commercial from Konica-Minolta. They are selling office equipment (having sold off their photographic equipment division to Sony), including an advanced copier. The commercial shows two office workers discussing the capabilities of their new Konica-Minolta copier, and one of them mentions the built-in biometric access facility. The other one responds with a warning that the machines are going to take over the world and take dominion over humans. The first office worker gives him a look that plainly says, ‘What a nut’, and turns away.
I think it says something when marketing for a major corporation here is willing to take a pot shot at this sort of paranoia.