Cloud and Devops: eliminating an entire layer of assholes

I've been in quite a few gigantic enterprises. Medical, Insurance, Retail, Government, you name it.

Without exception, in every one of these organizations the infrastructure teams function as blights upon innovation and adaptability, and impose unreasonable costs and delays. Infrastructure is not simple, I will grant... and this is not revelatory news to anyone in the industry.

It used to be, oh, say, a decade ago that we heard only rumors of Google, Facebook, Amazon, and other firms with dynamic, adaptive, cost-effective, anticipatory, and plentiful infrastructure for testing, prototyping, and production. No nightmares of 8 extra GB of RAM costing $1,000, or a simple PC server carrying $10,000 / year in upkeep chargebacks.

Once rumors, AWS / Rackspace / Google Compute and a host of other cloud providers have made that a much more stark contrast to anyone mired in internal infrastructure. Once things like Chef and other automated provisioning come into play, the contrast becomes even more pronounced.

Again, none of this is revelatory. Clouds are better cost, more adaptive, efficient, allow you to discover new technologies and make those practical to your job, group, and enterprise. Anyone who has done a cloud-enabled project has experienced the nirvana... But only now when we are being cast back into the rats nest of mandated internal infrastructure use do I recall all those meetings and emails and total clusterfucks:

• your RAM is too big, we can't support more than 8GB. Oh, you want 4GB? That will be $1000/year extra
• oh, you need another 100 MB of disk space, that will be $1000/year. No we won't provision you extra space, you have to crash in production and then raise a ticket to get us to do anything.
• Let's schedule 10 meetings to discuss minor hardware spec increases that should be 1/100 of the labor waste of the meetings
• Hardware guys bloviate about what your application's hardware spec requirements are, even though they have NO IDEA about what it does or its user base
• CPU? You like 386s right? 
• Oh, your VM is colocated with other extremely high use VMs? Too bad, so sad.
• SSD? What is that?
• Cloud? We have an internal cloud. No we don't have elastic storage or API provisioning. We can get your new VM up in 5 months rather than 1 year!
• Accounting... ah, the accounting. "It's free... if you can get it". "It has no upkeep cost, but it costs a ton to setup". "Will the cost drop over time as Moore's Law marches on? NO". 

I am so fucking sick of it. Thus I DECLARE:

The #1 Feature of Clouds is the Elimination of an Entire Layer of Assholes From Your Life

That's right. #1 feature by far. No one from our cloud provider has ever sent us threatening emails yanking our RAM size, or down-specced our machines without consultation, or dragged their feet to provide new machines, disk space, CPU, whatever. Or spent weeks ignoring tickets behind ticket walls. No one scheduling bloviating meetings and handing down arbitrary new policies from above. No meddling in your application architecture. No complaints to the CEO if you decide to use someone else's infrastructure. None of it.

Not just any arbitrary layer of assholes. Infrastructure lends itself to controlling and sociopathic shitheads. They get to ooh and aah over high-spec machines, they get large lump-sum budget line items, and, best of all, they get to lord over the distribution of those resources to the people that need them, and demand asskissing and tribute. This is a particularly obnoxious layer of entitled, intractable, petulant assholes you are extricating from your life.

What is the #1 downside of our collective profession? Stress. Stress that causes measurable, documented deleterious health effects. If you can identify something that significantly reduces that stress, and I argue clouds can do that, then it should be a priority: personally, professionally, and industry-wide.

Language Evolution Battle Royale.

So I read a lot of programming blogs and the inevitable debate about languages. Rust, Go, Java, Groovy, Ruby, Python, Javascript, C, C++, CSharp, etc.

What would be really fun is a long, thoughtful, crowdsourced, dynamic converging evolutionary practical language bracket shootout.

You start with a March Madness style of bracket. Not random, but the MAJOR languages seeded by their popularity.

Lesser seeds go to languages of note, usually in-use languages of lesser popularity with nonetheless some valid use case, but also emergent languages (Go, Rust), and some important historical ones not yet reproduced (Ada, APL, Smalltalk, ?Delphi?).

Lisp doesn't get in. If you've ever dealt with a lisper (how would you know... DON'T WORRY THEY'LL TELL YOU) you know they just aren't constructive members of any practical language discussion. Haskell is probably also not in for basically the same reasons.

For each language in an individual match, the features, standard library, runtimes, and debated. The outcome of each match is an evolved merged language... one that takes an effective compromise of the two languages, making sure to keep in mind the seeding of the sourced languages: if one language is bigger seeded, the reasons for its popularity/dominance can't be dramatically undermined in the evolutionary product.

Each evolutionary advancement in the bracket should note what was left out.

I (at the bottom of the blog, assuming bracket ninja stays around longer than next Tuesday) basically structured 16 languages. I only seeded the top four: Java and C++ are number one seeds, Javascript and Python are the #2 seeds.

From there, I didn't go by the usual ranking. Instead, the "competitors" for the immediate opponent and hopefully the second-round opponent deal with the primary domains of the language.

So C++ is obviously from the C family of languages and exists primarily as a systems language and, due to popularity and historical gravitas, is in many other domains. While Objective C isn't really a systems language, it is targetting devices with some aspects of performance and embedded problems. It also is a direct descendent of C as well. Objective C has strong recent popularity in TIOBE, but this is mostly due to the authoritarian dictates of Apple rather than a matter of public choice. Anyway, since these are both close in lineage and similarity, lets have them blast it out in the first round to produce a reasonable direct-from-C immediate evolutionary step

Similarly, Java is paired with CSharp. Both are essentially enterprise programming platforms, and quite similar in approach. So let's have them battle it out and produce the evolutionary result of managed portable C-esque languages with great standard runtimes and deep software ecosystems.

Python and Ruby are paired in the second-seed battle. Both are high-productivity scripting languages intended for both small and medium scale software packages, and dabble with aspirations of large-scale code. Both have their primary mission for fast-start web applications and the leading scripting languages. Both have worked hard to leave their scripting limitations behind with LLVM runtimes for performance and modern multithreading abilities.

The last seeded battle is Javascript and Dart. Javascript is seeing massive adoption rates due to the web (duh) and has made notable forays into other areas (Node.js, etc). Dart is the most obscure language on here with the least adoption, but is the most preeminent challenger to address JS's massive shortcomings as a fundamental language. Here also will likely be the battle to determine the fate of prototyping as a viable OOP feature moving forward.

It should be noted that most of the four top seed battles are really to address the glaring issues in the main four platforms.

The other four faceoffs all relate to the set-of-four subbracket they belong to.

 So Go and Rust are addressed at systems programming like C++, but much more exotic. The second round mashup between Objective++ and GRusto should produce a systems language par excellence.

Scala and Ada are two enterprise-focused languages (Ada has strong historical strength, once the #3 language in TIOBE). The eventual hybrid of Java# and Scalada should produce a featureful yet very performant and productive enterprise programming platform with strong typing and scalability.

Visual Basic and Smalltalk are two high-productivity languages with strong IDE designer components and RAD aspects, aligning with Python and Ruby high-productivity loose-typing applications. Smalltalk's legendary status as the ultimate IDE-supported OOP language needs to be represented. Visual Basic, despite the guffaws, has produced more applications in its programming for the masses than most non-top-seeded languages. The outcome of BasicTalk and Rubython should produce an amazing high-productivity scripting application building capability with phenomenal database integration and UI construction from desktop to web.

Finally, there is Perl and PHP. PHP is too popular to not include. Perl is too popular to not include. Both are viewed as intermediate steps in the evolution of their domains, and remain very heavily used. PHPerl should cull whatever is good about these platforms not that was missed by their successors, be it simplicity, frameworks, syntactic sugar. Combined with JavaDart in the second round will produce a useful, flexible language with unique programming constructs and a powerful standard library with nods to other domains such that it can expand into other places. This overall bracket represents the "ugly but effective" group of languages and platforms that produces an enormous, undeniably proportion of modern software and can't be ignored.

From those four domains, combined evolutions may be harder.

What is left out? Well, the free bracket only allowed me 16. I think that Ruby/Perl should be subjected to a play-in game before facing Python, or strangely Python/Perl face off. Ada should probably be subjected to a play-in bracket of many other languages in general systems use from the ages.

The lack of R/APL/Fortran and other heavyweight array/matrix/stats/math processing should probably be inserted into the systems sub-bracket. Go and Rust can battle it out in a play-in prelim, and then incorporated with the powers of the raw computing languages.

Scala should be representative of a grand JVM language battle between groovy, scala, and all the others.

In the end, we see if some comprehensive language can be ideated that isn't the tragedy of the commons. 

Stoopid Idea of the Day: Five lines or less

Oh hey, an interesting StackOverflow question:

http://stackoverflow.com/questions/811074/what-is-the-coolest-thing-you-can-do-in-10-lines-of-simple-code-help-me-inspir

Of course it was closed as unconstructive.

How about five lines? One hand, one statement per finger, if you need that.

Pointer finger: demonstrably state the input instruction/line

Middle / Ring / Pinky: demonstrably process the input

Thumb: demonstrably output or return (ok, if you really need to do some processing here in the return...)

Line count doesn't include basic stuff like function bodies, class overhead, imports, etc. 

Also, state language, environment/os/libraries needed.

If your "line" is longer than, oh, 120 characters, then is that really a line? As in, if you are a master of obfuscated C/C++, please, just don't...

APL is also cheating. 

Heavy UNIX commands... well, I guess everything is relative...

People: readable five lines of code that perform a useful, adaptable, extendable purpose.

Some good tagging and relations plus moderation... 

Yet another build system? Please?

Problem: Most gradle/ant builds turn into groovy/XML soup

Problem: Gradle/Ant projects can lack a good idea of "where to start" to build something. What properties do I need to define? What order of tasks? What do the tasks depend on (db, etc)?

Problem: Maven is a PITA and inflexible, and the advocates are tyrannical

Problem: Maven still ends up with a massive huge xml file

Problem: Maven won't do simple, literal directory-of-jars

Problem: Maven's internet connection dependence

Problem: If maven dep-resolve fails you (IBM Websphere, another reason to hate you), you're basically screwed.

------------------------

What are the good parts that have been invented:

- (Ant) a general set of plugins for templating/replacement, files, building code, etc etc.

- (Ant) decent IDE support

- (Maven) maven repository: the failed dependency resolution is a rare case. Generally, this thing is a very good thing for code sharing and library use/reuse. Gotta leverage this massive body of work and mindshare.

- (Maven) standard project structure: although src/main/java isn't my favorite thing (three levels down? really?), the overall maven project structure provides acceptable structure that people are used to now.

- (Maven) standard project structure for WARs: again, I am not too excited about the file structure, but it'll do.

- (Maven) standard jar naming <artifact>-<version>.jar, a great thing indeed

- (Maven) binary / javadoc / source jars

- (Gradle) groovy scripting for those things you need scripting

-----------------------

What I want from a build system

-  leverage the structuring standards of Maven that have made unfamiliar projects more straightforward to interpret

- leverage Maven repos and dependency maps, but add flexibility fallbacks for literal jars, and only do dependency resolution when new projects are setup or new libraries/dependencies added (one-off or one-time tasks)

- this is a new feature for java builds: a single file download that specifies where to download the rest of the project (sourcecode, etc). The build system will connect to sftp/ftp/git/subversion/whatever to get the code, then do dep resolve. Kind of like .rpms for linux. Great way to do starting points for projects, especially ones without a fancy site or github...

- use some of the nice features that make Ruby on Rails so good: convention over configuration, and folder structure as configuration. Maven failed at this since you end up with a huge POM xml. 

  - Why not have a naming convention for mulitple source folders: as long as they match /src-* and are at the project root, then they are an additional source folder. 

  - /info dir has the project info. 

  - /lib for various libraries. In the /lib, you can have .jar files for inline/literal inclusion, or an empty file named <artifact>;<version> that triggers dependency resolution. Nice instant  visual indication of the libraries used to build a project. Subdirs for main for main build, test for test build and execute, etc.

  - /tasks is a dir with more subdirs <taskname>, and each of those has their own gradle/ant script that can be invoked

Need to add a quick jar? drop it in the /lib/main dir. Need to add a quick set of dependencies from another project that you know works? Copy-paste the dependency files into the lib dir. Source tree getting a bit thick in the /src/main/java folder? spin off some more src-* source directories. Maybe even tasks can become drag-and-drop or copy/paste with sufficient automation/decoupling. 

Web resources, ears, subprojects, project relationships, etc: all to be determined. 

Another key aspect should be IDE support and multiple IDE usage. perhaps a separate directory tree for ide-specific settings. Ant/Gradle/Maven xmls can be autogenerated for IDE import/usage if need be. 

Project structure:

/	root of project
/lib	libraries and dependencies
/lib/main	libs and deps for main compile
/lib/test	libs and deps for unit test exec
/src	maven-style primary source dir, with /main/java, /tast/java, /main/resource, etc
/src-	more maven-style source dirs, such as a service impl, etc
/task	nonstandard task scripts

Frameworkitis and False Enterprisey Security: WS-Security and signing.

http://blogs.msdn.com/b/vbertocci/archive/2005/04/25/end-to-end-security-or-why-you-shouldn-t-drive-your-motorcycle-naked.aspx

Who wears a motorcycle helmet indoors? Just because it's a simple analogy doesn't mean it's right.

This is still rampant frameworkitis. HTTPS isn't good enough! What happens if something is done after the message reaches it's point?

Oh, you mean the fucking ENDpoint?

99% of SOAP bloat systems (aka, licensed application servers configured and installed by consultants) IMMEDIATELY deserialize a SOAP message to a datastructure, and from there that data structure will be mangled by database call or MQ message or whatever.

Notice the uselessness of message signing. It's a false security blanket sold to stupid fucking executives. The data will be plaintext in a host of different settings post-parse: SQL requests, followup messages. That requires either good internal data practices or specifically tailored security architecture.

READ: less than 1% of cases, and outside the concerns of SOAP, which is a messaging protocol, not a transport protocol.

It's frameworkitis and enterprisy. Let's add a bunch of stuff that isn't the concern, bloat, bloat, bloat, million dollar license, buy a ferrari.

Fucking software companies. 

A little Spring XML to JSON conversion experiment...

What might some random Spring config (in this case a Spring-batch example) look like if JSON were the configuration/injection message format? What is good, what is bad?

 <bean id="taskletJob" parent="simpleJob">  
   <property name="steps">  
     <list>  
       <bean id="deleteFilesInDir" parent="taskletStep">  
         <property name="tasklet">  
           <bean  
       class="org.springframework.batch.sample.tasklet.FileDeletingTasklet">  
             <property name="directoryResource"  
               ref="directory" />  
           </bean>  
         </property>  
       </bean>  
       <bean id="executeSystemCommand" parent="taskletStep">  
         <property name="tasklet">  
           <bean  
       class="org.springframework.batch.sample.common.SystemCommandTasklet">  
             <property name="command" value="echo hello" />  
             <!-- 5 second timeout for the command to complete -->  
             <property name="timeout" value="5000" />  
           </bean>  
         </property>  
       </bean>  
     </list>  
   </property>  
 </bean>  
 <bean id="directory"  
   class="org.springframework.core.io.FileSystemResource">  
   <constructor-arg value="target/test-outputs/test-dir" />  
 </bean>  

My swing at JSON, using JSON-ish features and advantages (lists, maps/objects, etc)

 {  
   "taskletjob":  
     {"#t":"BEAN", "id":"taskletJob", "parent": "simpleJob",  
     "properties": {   
       "steps": [  
         {"#t": "BEAN",  
          "id":"deleteFilesInDir",   
          "parent":"taskletStep",  
          "properties":  
           {"tasklet":  
             {"#t":"BEAN", "class":"org.springframework.batch.sample.tasklet.FileDeletingTasklet",  
              "properties": {"directoryResource" : {"#ref":"directory"}}  
             }  
           }  
         },  
         {"#t": "BEAN",  
          "id":"executeSystemCommand",   
          "parent":"taskletStep",  
          "properties":  
           {"tasklet":  
             {"#t":"BEAN", "class":"org.springframework.batch.sample.common.SystemCommandTasklet",  
              "properties": { "command": "echo hello", "timeout": 5000}  
             }  
           }  
         }  
        ]  
       }  
     },  
   "directory": {  
     "#t":"BEAN", "id":"directory",   
     "class":"org.springframework.core.io.FileSystemResource",   
     "constructor-args": ["target/test-outputs/test-dir"]  
   }    
 }  

So what I found out:

Size in LOC was mostly identical. No real big savings there.
Spring utilizes namespaces implicitly or explicitly... No real JSON mechanism, so I invented the "#t" which is supposed to indicate the object type if it is typed, to give guidance to some parser/bean construction code as to required fields, etc.

In other places, like properties, the JSON object is used more as a map, since it is more intuitive that way. The JSON format here seems much more natural, and seem to save lots of keystrokes. I mean would you rather type:


<property name="command" value="echo hello" />
<property name="timeout" value="5000" />

or

"properties": { "command": "echo hello", "timeout": 5000}

Clear win for the JSON there.

However, JSON has some weird corner cases in formatting, such as arrays of objects. How do you render that without wasting extra tabs or lines? Granted, the <list> construct in spring xml wastes a tab and a line...

And all the double-quotes in JSON. Annoying. Not as annoying as xml ending tags. And why are commas even needed?

Other JSON problems: Getting lost in curly-brace land. WHile annoying verbose, at least closing tags in XML let you see what's being ended/closed. But considering most modern editors have curly brace features for C and Java code, this isn't a huge deal.

But if I had my druthers with a variation on JSON, where you don't need double-quotes on the object keys, values are optional, and drop the commas, why can't JSON be this?

 {  
   taskletjob:  
     {#BEAN id:"taskletJob"   
     parent: "simpleJob"  
     properties: {   
       steps: [  
         {#BEAN" id:"deleteFilesInDir"  
          parent:"taskletStep"  
          properties  
           {tasklet:  
             {#BEAN class:"org.springframework.batch.sample.tasklet.FileDeletingTasklet"  
              properties: {"directoryResource" : {"#ref":"directory"}}  
             }  
           }  
         }  
         {#BEAN id:"executeSystemCommand"  
          parent:"taskletStep"  
          properties:  
           {tasklet:  
             {#BEAN "class":"org.springframework.batch.sample.common.SystemCommandTasklet"  
              properties: { command: "echo hello" timeout: 5000}  
             }  
           }  
         }  
        ]  
       }  
     }  
   directory: {  
     #BEAN id:"directory" class:"org.springframework.core.io.FileSystemResource"  
     constructor-args: ["target/test-outputs/test-dir"]  
   }    
 }  

... Not a huge difference, but it would make reorganizing and moving object chunks around without worrying about breaking the commas in a list, or the fields inside an object. I think the parser would be about the same, no impacts to speed. But JSON the format spec, that ship has sailed... Maybe it looks too much like S-expr...

Oracle WINS!!!!.... what?

So Oracle has won... something.

I'm scared that Google has won this with a spectacular Oracle pyrrhic victory. But we as software developers may be losing in a big way.

If Oracle gets to copyright a basic rangecheck function, things are going to get really annoying.

I wish I could talk to a copyright judge and make this argument:

"Can you copyright the word 'is'?" (I assume no)

"Can you copyright the words 'greater than'?" (again I assume no)

"Can you copyright the words 'greater than this but less than that'?" (again I assume no)

And really, right there, you have pseudocode for rangecheck. For the java version, well, basically, you're just with a dialect translations: "Can you copyright the words 'superieure a cette mais inferieure a celle'?" (French google translate, and again I assume no).

I mean, it's not even a quote. There is no depth of meaning, or abstract reference, or hidden commentary. It is a direct request statement, basic functional language. How can that be copyrighted?

I mean, imagine the lawsuits that come out of this. Patent armories don't help for defense. Only writing tons and tons of library code and copyrighting it. You might have tons of excess software written just so they can sue other people for doing similar stuff.