2013.11.11 – Waldo just posted The State Decoded 0.8 Release. This is a *huge* update that we’ve spent the last few months working on. 577 changed files, 127,076 additions, 5,123 deletions. That’s a lot of code. There are a few pieces I would have liked to squeeze into this update, like abstracting the XML import to make JSON importing more user-friendly, and cleaning up the admin system – but those will come for the 1.0 release. Which is pretty close on the horizon. In the meantime, check out the 0.8 release of State Decoded on Github!

Read This

2013.11.01 – I ran into an issue with a Solr configuration that was working for me locally, but not on our CentOS 6.4 server. I’ve documented below all of the issues I encountered along the way, as the upgrade from Solr 4.2 to 4.3+ is a pretty nasty one, due to major changes in the logging system (LOG4J / SLF4J)

Lucene/Solr 4.3+ : The Beginning

java.lang.IllegalArgumentException: No enum const class org.apache.lucene.util.Version.LUCENE_43

<luceneMatchVersion>LUCENE_43</luceneMatchVersion>

org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Plugin init failure for [schema.xml] fieldType "descendent_facet_hierarchical": Plugin init failure for [schema.xml] analyzer/tokenizer: Error instantiating class: 'com.o19s.RegexPathHierarchyTokenizerFactory'

Maven

cd;
wget http://apache.tradebit.com/pub/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.tar.gz
tar -xzf ./apache-maven-3.1.1-bin.tar.gz -C /usr/local

sudo ln -s apache-maven-3.1.1 maven

sudo vi /etc/profile.d/maven.sh

export M2_HOME=/usr/local/maven
export PATH=${M2_HOME}/bin:${PATH}

JDK

sudo yum list jdk
sudo yum install java-1.6.0-openjdk-devel

sudo vi /etc/profile.d/java.sh

export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64

Upgrading to Solr4.3+

A note on Tomcat logging

less /var/log/tomcat6/localhost.[DATE].log

/etc/tomcat6/Catalina/localhost/solr.xml

<Context docBase="/opt/solr/solr.war" debug="0" crossContext="false" >
   <Environment name="solr/home" type="java.lang.String" value="/opt/solr" override="true"/>
</Context>

Now back to the good part

wget http://mirrors.gigenet.com/apache/lucene/solr/4.5.1/solr-4.5.1.tgz
tar -xzf ./solr-4.5.1.tgz

sudo mv /opt/solr /opt/solr-4.2.1

sudo cp -R ~/solr-4.5.1/example/solr /opt/solr-4.5.1
cd /opt
sudo ln -s solr-4.5.1 solr

cd /opt/solr
sudo cp ~/solr-4.5.1/dist/solr-4.5.1.war ./
sudo ln -s solr-4.5.1.war solr.war

SEVERE: Exception starting filter SolrRequestFilter
org.apache.solr.common.SolrException: <strong>Could not find necessary SLF4j logging jars.</strong> If using Jetty, the SLF4j logging jars need to go in the jetty lib/ext directory. For other containers, the corresponding directory should be used. For more information, see: http://wiki.apache.org/solr/SolrLogging

sudo cp ~/solr-4.5.1/example/lib/ext/* /var/lib/tomcat6/lib/
sudo cp ~/solr-4.5.1/dist/solrj-lib/* /var/lib/tomcat6/lib/

sudo cp -R ~/solr-4.5.1/dist/* /opt/solr/lib/

sudo cp ~/solr-4.5.1/dist/solrj-lib/* /var/lib/tomcat6/webapps/solr/WEB-INF/lib/

sudo vi /opt/solr/solr.xml

<cores adminPath="/admin/cores" defaultCoreName="collection1" host="${host:}" hostPort="${jetty.port:}" hostConte    xt="${hostContext:}" zkClientTimeout="${zkClientTimeout:15000}">
    <core name="collection1" instanceDir="collection1" />
    <core name="baltimorecode_dev_www" instanceDir="baltimorecode/www/staging/statedecoded" />
</cores>

Finally Rebuilding the custom Solr tokenizer

cd /opt/solr/baltimorecode/www/staging/statedecoded/src/

sudo mvn package

cp target/regex-path-tokenizer-0.0.1-SNAPSHOT.jar ../lib/regex-path-tokenizer-0.0.1-SNAPSHOT.jar

sudo service tomcat6 restart

lynx http://localhost:8080/solr/baltimorecode_dev_www/admin/ping

ssh -L 8009:localhost:8080 user@removtehost.org -N

Finishing Up

1. Patrick Reilly: How to install maven on CentOS
2. Solr: MultiCore
3. Apache Tomcat configuration: HTTP Connectors
4. Andrew Jaimes: Installing Lucene/Solr on CentOS with Tomcat
5. Hipsnip Cookbooks issues: Solr Fails to start missing SLF4j
6. StackOverflow: Solr 4.3 Tomcat6 Ubuntu installation exception (More confusing than helpful.)

Read This

2013.10.21 – A few months back, there was a rather interesting discussion on Reddit about different internet technologies. I put the following comment together as a bit of historical perspective.  In retrospect, it’s probably not 100% accurate, and a bit or a rant, but I figured other people might find it interesting. Your webserver back in the old days was probably NCSA HTTPd if your host hadn’t switched over to Apache. Apache is still one of the most popular webservers today. NCSA HTTPd, not so much. Apache is written in C, but you don’t need to know any C to configure it. Webservers generally deliver files, most often HTML which is just a marked up text file, but also images and videos and audio and pdfs and anything else you want. A few years later and you also commonly see Nginx as a webserver and Microsoft’s IIS on Windows servers. Programming on the web pretty much started with CGI scripts and was purely server-side. Back in the early nineties, this is how we did things – generally a C program that was compiled, and set to receive input from the webserver and do something with it – your form emailers and counters and guestbooks and so forth. You really don’t see much of this any more. Most people switched over to Perl as an easy way to write scripts, rather than using compiled executables. Perl is notoriously hard to read and write, but a pretty good language for string processing (which is basically all the web is at the core). It has declined in popularity significantly over the last ten years or so. Java began to show up more for web use towards the end of the nineties. These days, it’s synonymous with “Enterprise”, and generally used for large business applications. It’s a very dense language, and lots of frameworks exist for it, including ones like Spring, Hibernate, Struts, etc. Many, many other languages have grown up around Java, including Scala, Clojure, and Groovy. PHP showed up about then as well, and quickly became the most widely-used language on the web. This was primarily due to its friendliness, ease of use, and ease of deployment – you could just drop scripts on a server the same as with html files and run them. Since PHP is so easy to use, lots of hobbyists have picked it up, and a large amount of awful code and poor tutorials have been written with PHP. Even so, the changes since version 5 have been increasingly bringing PHP towards the modern era, and it’s actually quite good these days. (Haters gonna hate.) There are many frameworks for PHP, including Symfony, CodeIgniter, Laravel, Cake, Zend, and others. Several of the most popular Content Management Systems (or CMS) are written in PHP, including Drupal, WordPress, and Joomla. Ruby was gaining steam by the early 2000s. Based off of Smalltalk, it’s a very “smart” language that’s actually a lot of fun to write. It’s got a lot of niceties that do help you write code with less effort. The joke is that hipsters use Ruby, but it’s really a great language. It also has a fantastic package manager RubyGems / gem. Rails is the most popular framework for it; most other web frameworks often tend to just be clones of Rails. It’s a very opinionated framework, though – with lots of, what we call in the programming world, “magic”. Another popular framework is Sinatra, which is a thinner framework. Python has technically been used on the web as long as PHP, but still doesn’t have very much marketshare. It is, however, used for not-web things frequently, especially among academics. In some schools, it has replaced Basicas the first language to teach children, due to its friendliness. The most popular framework is Django, though there is also Pyramid/Pylons and Flask. ASP also needs a mention, though it is not an OpenSource language like the others, and owned by Microsoft. You often see it in conjunction with .Net, and it’s very popular for use in large corporations (pretty much anywhere you might otherwise see Java). It’s not very friendly, IMHO. In the last few years, some older “purely functional” languages have come to popularity as well, including Haskelland Clojure (outgrowth of Lisp). Now, you still needed somewhere to store all of your data, so the first popular databases came about. Most of these were ANSI SQL variants, and so behave almost identically, with the main differences being under the hood. These include: MySQL : Still one of the most popular databases in use on the web. There’s a popular fork known as MariaDB as well, which gives you a few new tricks (like server clusters!). PostgreSQL: probably the second most popular. (I’m guessing.) MSSQL: the obligatory Microsoft version. and SQLite which actually just operates on text files and is super lightweight. And last, there’s Oracle which is a megalithic company all to itself, as well as a database, and a Business Intelligence service provider. Large organizations, colleges, and universities give them lots of money to make their systems go. And then you have the no-SQL data stores that came to popularity in the last ten years or so, which generally are non-relational document stores. There are lots of these, but a few of note are: Memcache was one of the first and most popular, especially since Facebook picked it up. MongoDB which gives you a lot of the abilities of a relational DB.  Mongo has recently received a lot of negative criticism due to fundamental flaws in implementation.  [1] [2] [3] CouchDB another Apache project that uses JSON to store data. Cassandra yet another Apache project, designed for high-availability of large amounts of data Tokyo Cabinet and Kyoto Cabinet successors of dbm. Now, going back for a second, we eventually wanted browsers to be able to do some neat things, so Javascriptcame about – a programming language that runs in your browser. It actually doesn’t have much in common with Java, regardless of the name, and is actually an outgrowth of ECMAScript, which also gave us ActionScript which is used in Adobe’s Flash multimedia platform. There are many, many Javascript frameworks. The old bunch, which mainly made writing code a bit easier and doing things like animations, included Prototype, Scriptaculous, and jQuery, and newer ones such asUnderscore, and Zepto. The new bunch is are more similar to traditional web frameworks, giving Models and Views (and sometimes Controllers or otherwise just Routers), including Backbone, Ember, Knockout, and Angular. In the last few years, Node has appeared, as a way of using Javascript on the server-side. It’s actually very performant for IO operations, but suffers from being relatively young, and people are still finding the best ways to use existing frameworks with it.  Still, it provides an excellent package manager called NPM which is ver similar to Ruby‘s.

Read This

2013.09.23 – I’ve been working with our local Legal Aid office for most of my career – their website was one of the first ones I ever worked on, in fact. Over the last few months, I’ve been working with Charmed Designworks to create a new site for LAJC. Behind the scenes, we’ve switched over to WordPress and are using some really nice <a href=”http://sass-lang.com” SASS CSS Preprocessor”>SASS</a> + Compass to make styling a breeze. Legal Aid Justice Center new website

Read This

2011.03.20 – Let’s say you’ve already gotten through the basics of JMeter and you’re ready to start setting up your testing. If you’re doing any sort of remote testing, you’ll inevitably need to know how to setup your client/server relationships. The vast majority of JMeter’s configuration is done through a single file, the jmeter.properties file (which lives inside of the JMeter bin/ directory). Any of the properties in this file can be overridden by options on the command line – but since this is Java we’re talking about, the method is ridiculous (code for Linux/Mac): <br /> jmeter -J<em>propertyname1=value1</em> -J<em>propertyname1=value1</em><br /> Of course, that doesn’t work for every option. For instance, to tell the server that you want it to listen for the initial request on a port that’s not random, you have to do so as an environment variable set on the same line before you make the call to the server. Truly and utterly bizarre: <br /> SERVER_PORT=1660 jmeter-server<br /> As noted below, the server_port only sets the port used for the initial request from the client to the server to begin testing, the response from the server to the client will be sent on a totally random port. I also found that the setting above did not work for setting the listening port at all!

Server (Slave) Configuration

Client (Master) Configuration

Read This

2011.03.19 – Recently, I’ve been doing a bit of load testing on Amazon AWS after reading cloud storage reviews to determine how much abuse our web application can take without killing the server. I’ve been attempting to use Apache JMeter to do the hard part, but came up against a slew of problems. The documentation provided seems targetted at dyed-in-the-wool Java developers (that “J” at the beginning is clearly a warning shot), and makes pretty big assumptions about the knowledge of the audience. Here are the basic concepts of how to get started using it, targeted for us LAMP developers. The first thing to understand is that there are two main to go about using Jmeter. By default, Jmeter runs as a free-standing (GUI) application on which you run Tests directly from the machine it’s running on. You do this with the Run > Start option. You can also, however set it up to run on other machines, reporting the results back to the original GUI. In JMeter terms, the Master from which you send the tests is called the Client, and the Slaves that run the tests are the Hosts. You have to configure which hosts to run on – afterward you can use Run > Remote Start > <em>slave address</em> to run the test on a single machine, or Run > Remote Start All to run on all slaves. To get started, try running JMeter on your local machine, and writing a basic test. If you’re working locally, just running the bin/jmeter (or bin/jmeter.sh on Mac, bin/jmeter.bat on Windows) script should start up a java session and run the program on your machine. If you’re working remotely, Jmeter runs in an X-Windows environment – so if you’re on a Mac you’ll need to have X11 installed and running.

Writing Tests

• A Thread Group. Used to set the number of virtual users (Threads) and number of Loops (iterations, obv) for *each* slave to perform (if you’re using the local machine) Click on the Test Plan and then chose Add > Threads (Users) > Thread Group from the Edit menu or the right-click contextual menu. For the trial run, you might just give it 1 User, 1 for Ramp Up Period and 1 Loop
• A Listener of some type. This is the what shows you the results of the test, either by a chart, table, or other medium. The simplest one is probably the Summary Report – to add it, click on the Thread Group and choose Add > Listener > Summary Report from the menu. No additional configuration is necessary for this type of Listener.
• A Sampler – an actual test element. For instance, if you want to just grab one page off of a site to see if it’s working, you’ll add an HTTP Request Sampler. Click on the Thread Group again, then choose Add > Sampler > HTTP Request. We’ll want to test with a site we know is working first, so enter google.com for the Server Name or IP.

Read This