Data and the Web

Well, it's the last day of Web 2.0 Expo. If this turns out like the the first two days of the expo, it should be another great day of continuous conversations and demos.

Web 2.0 Expo is certainly a much larger show than the MySQL conference. There's simply an enormous number of people here, so the dynamic is quite a bit different. When we were at the MySQL show, it felt like hanging out with a close group of friends; at Web 2.0 Expo, it's more like being at a rock festival.

Although the scale is different, one constant is the amazing new technology that's on display. One of the eye-catching tools that we've enjoyed seeing at both shows is MindTouch's Deki Wiki. This is basically a next generation Wiki platform that really expands the idea of what a Wiki can be. For instance, it has both an API and the ability to host structured data. This is exciting for us, since it opens up the possibility of creating a “Data Wiki”– a platform for publishing and editing data in a collaborative way. Maybe the idea of a “Wikipedia for data” is not too far away after all.

Overall, it's certainly fun to be in the middle of all this. At just about every moment we've had a lots of people gathered around our booth and it's been really great to be able to demo Kirix Strata™ in person. I'm sure I'll probably be beat by the time I get home, but at the moment, I'm just fired up just to be here. Now, if I had only brought that extra box of business cards along…

During our journey from the MySQL Conference last week to the Web 2.0 Expo this week, we've continued to see a common pattern. There are a tremendous number of back-end business systems and almost all of them seem to rely on the humble Web browser for the front-end.

Now there is a very good reason for this — the Web is a tremendous distribution platform and companies are taking advantage of the same protocols and tools on their intranet that have already propelled the growth of the Internet. And, bottom line, browsers are really, really easy to use.

Data Integration 2.0

As an example of this situation, we met some folks last week from Snaplogic. Their product makes it really simple to integrate data from back-end systems and enables you to access data from anywhere, mash it up and publish it. However, when they were demoing the software, there wasn't particularly much to show off, since a normal browser can't do much with the data except let you “Save As” to your desktop.

Thankfully, this type of front-end work is perfectly suited for a data browser. So, we decided to demo each other's products at the Web 2.0 Expo this week. Snaplogic is able to easily show the value they bring to the table by actually displaying the data integration they can do. We put together a quick dashboard with Snaplogic data and were able to show off all the cool stuff you can do once you've got a data feed to work with. Indeed, this was like the serendipitous combination of chocolate and peanut butter.

The Last Mile

This reminded me of a great metaphor used in a post by Juice Analytics called “The Last Mile of Business Intelligence.” The idea is that businesses have built lots of large back end systems — from data integration and warehousing to BI and reporting systems. But, in the end, it still seems that actual users are left out in the cold. The “last mile” between the heavy machinery and the actual knowledge worker has not been adequately bridged.

So unfortunately, the typical pattern involves working with a back end system via a Web browser and then, ultimately, exporting a CSV file to work with in Excel.

This is not terribly efficient. A data browser enables you to open up those same intranet pages and perform your normal browsing tasks. However, because it's a data analysis tool at heart, you can actually open up data sets and start working with them directly. Then you can do fun things like bookmark tables, build calculations, create reports and write queries and have it refresh the next day with the latest data. Because of this, it plays very nicely in the worlds of of exploration and reporting per Juice's “data value framework.”

Why Incorporate the Web?

In the past we've shown off some of the fun things Kirix Strata™ does — like open up HTML tables natively or showing RSS feeds in a table view — but often we'll get a response like, “that's cool, but what do I do with it?” There is no question that tools like these can be very useful for some tasks (e.g., integrating data from the web with local tables), but they're really just icing on the cake.

The important thing is how tightly the web is integrated with the data analysis tool. This opens up all kinds of possibilities for direct interaction with both the Web infrastructure and the database systems that power business intelligence in the enterprise. Add some Web APIs into the mix, and now you could have access to the contact information in your ACT database, the corporate newsletter contacts in a MySQL system and all the recent sales information from your Salesforce.com account. Explore and integrate at will.

Given that the meme “Enterprise 2.0″ resounds louder each day, we should only expect to see Web infrastructure reaching deeper and deeper into organizations. In the end, maybe part of the BI “last mile” solution is just giving businesses a browser with a bit more intelligence.

Just finished up showing Kirix Strata™ off at the MySQL Conference and Expo in Santa Clara, CA the past couple days.

We had some great crowds, excellent questions and plenty of discussion about how Strata can fit in to a MySQL shop. Besides the hands-on data analysis and querying Strata allows, people seem pretty impressed with how quick and easy it was to generate ad hoc reports.

We also had the chance to meet a lot of great people from various companies — definitely some pretty cool technology out there. One company in particular that we were impressed with was SnapLogic. They're sort of like Yahoo Pipes for the Enterprise — they make data integration really simple. We even had the chance to hook Strata up to some of their data feeds… it was like a match made in heaven.

Well, nothing like doing demos all day for two days in row. We'll try to match that and then some at the Web 2.0 Expo next week from April 23rd through the 25th. If you'll be attending, please make sure to drop by and say “hi”.

We're pleased to announce that Kirix Strata™ is now officially out of beta! A lot of hard work and late nights of coding have been put into this li'l tool and we hope it makes people's lives a little bit easier when it comes to data analysis.

Thanks to everyone who has been involved in the beta process — we'll get those free licenses rolled out to you within the week.

Also, you'll notice a big ol' website redesign too. We want to give a special shout out to Jeff, Benni, Peter and David for the bang up job they did with the design and implementation of the website. And, here's our new overview video, check it out:

(And here's an embeddable YouTube version…)

Now that Strata is out, we're looking forward to adding a lot more content to the site, in terms of more data-centric blog posts, product applications, tips & tricks as well as some open-source projects that we've been working on in the background. Oh yes, we also have a load of extensions that we are planning to release too. So much to do, so little time.

Thanks again to everyone. Please download the full version of Strata and try it out for 30 days free. If you have any questions, feel free to post a message to the forums or shoot us an email.

Great post today by Bret Taylor, seeking a utopian wikipedia for structured data. There are currently various attempts at this type of thing; Freebase, in particular, comes to mind.

But what Bret is talking about is less about the semantic web — where all data everywhere is linked together by certain tags and/or infrastructure — and more just about getting access to really useful chunks of information, like mapping data and white pages. Heck, it'd be great to just get all US area codes with related information in a nice, accessible CSV file.

Hard to know if this sort of thing could ever come to pass, given that there are likely far fewer people who would edit a large financial data table than, say, the latest goings-on in American Idol — however, I'd love to see it happen.

In the meantime, ReadWriteWeb has compiled a bunch of data sources that I thought might prove useful to you as well. Some of this stuff was new to me and I look forward to exploring them. The one thing that the article didn't mention was the great publicdata tag archive still happening on Delicious that we talked about many moons ago. Make sure to check that out too if you are looking for some good public data sets.

After a long, long beta cycle, we're pleased to finally announce a release candidate! Feel free to download it here.

This version takes Beta 9, released last week, and tightens the screws. We've done a lot of bug fixes and added plenty of little things here and there, like cleaning up some GUI items and making our nomenclature consistent. Oh yes — exciting things like determining if a menu item should be “Bookmark Location” or “Bookmark This Location” or if a dialog should be called “New Bookmark” or “Add Bookmark”. Yes, I can tell you're extremely enthused.

We've also tightened up the report writer functionality — the grouping wasn't working very well in Beta 9, but in this version, you can have multiple groups in your report. Also, we added the ability to create reports directly from queries instead of just tables. And, try out Quick Report: highlight a few columns in a table then select New > Report and see what happens.

And, drumroll please…

We finally have a help manual. It is a not-yet-final-draft, so please excuse any grammatical errors. It'll give you all kinds of extra information about doing data analysis using Strata that you probably didn't know existed. Select Help Contents from the Help menu and have a look.

We expect a second release candidate by mid-next week that should have everything tightened down even further… so stay tuned for more good stuff to come.

Please keep the feedback coming, it's really appreciated! Thanks!

We're happy to announce that Strata beta 9 is now available for download. This build includes tons of bug fixes and really tightens down the interface. Here are some of the more significant items in this release:

Formatting

We've had a long, winding history in trying to clearly present formatting options to a user. Formatting is mainly used in report creation, but it can also be used in a table view, for such things as coloring columns. Bottom line, there were lots of different options on the table.

In the end, we think we came up with a pretty good solution. By default, the formatting toolbar is now embedded in the report writer document view. This makes it really easy to do formatting in a report but also tucks away the formatting while using other parts of the package.

However, we really didn't want to limit the formatting to only reports, so we have an option in the View menu to show the Format Toolbar. If you turn this on, the Format toolbar will appear underneath the Boomkark Toolbar and will no longer appear in the report writer. It boils down to a toggle: either always show the format toolbar or only show it when creating reports.

So far in our testing, it seems to be working out well.

User Interface

There are lots of very small items here. We've tried to tighten down the UI as much as possible with this beta. We've updated some icons, worked on a consistent nomenclature in menus and dialogs and did the work on the formatting toolbar mentioned above. There may be some other small changes here and there, but this should be pretty close to what the interface will look like when we officially release.

Report Writer

There's been some really great development on the report writer front. We've got a new grouping/settings panel, which should make it easier to quickly define your page layout and groups. Also, many bugs are getting squashed.

However, the big feature to fall in this version is Save to PDF. You can now create a report and go to File > Save As External to save the report to a PDF file for archiving or sharing. It is never easy to get things working with external format types, but we're delighted to see this available.

There are still a bunch of bugs in report writer that will be sorted out very soon; we appreciate your patience.

Options/Preferences

Our Options/Preferences area (Tools > Options) has gotten a big overhaul. We now have settings for popup blocking, cookie management as well as script and report preferences. There are still a few things yet to be hooked up (the Privacy tab in particular), but these will be coming online very soon.

Scripting: DOM Bindings!

We've now got DOM bindings in our scripting, which will let you programmatically interact with the web control in the scripting language and let you design forms and controls that use web widgets.

We're been craving these bindings since we originally incorporated the web browser into Strata, as it really provides a ton of power to manipulate the web and build extensions.

Stay tuned over the next month for extensions and script examples that utilize these new bindings. We're looking forward to see what ideas everyone can come up with too.

Thanks for all the feedback; it is very helpful. Please keep it coming!

Boy, time sure flies when you are trying to finish a beta cycle…

I just wanted to give everyone a heads-up that Kirix will be showcasing Kirix Strata™ at two big tech conferences during the month of April:

MySQL Conference & Expo
April 14-17, 2008
Santa Clara, California
Booth #212

Web 2.0 Expo
April 22-25, 2008
San Francisco, California
Booth #244

If you happen to be attending either conference, please stop by and say hello. We'd love to meet you and chat about, well, pretty much anything. Our guys are known to provide completely off-road software demos too… so don't shy away from bringing along any crazy ideas.

See you then!

Hello everyone,

We're happy to announce that Kirix Strata Beta 8 is now available for download!

There's been lots of good work on Strata over the past month or so, with much of it focused on bug fixes and other tweaks for a better overall user experience. That said, here are a list of some of the bigger things we've tackled for Beta 8:

Data Conversion

We continue to try and make opening, saving, connecting to and integrating data as seamless as possible. We've taken another good step in this direction by enabling “Save As” to work between external database systems and external folders. So, for example, now you can do things like connect to a MySQL database and drag a table from there into an Oracle database system. This basically lets you copy tables from one system (or local folder) to another. And, of course, you can also do this same type of copying operation with data files already saved to your Strata project.

User Interface Enhancements

• We've done a lot more work on the Bookmarks toolbar. We're also delighted to finally have favicons in, which makes things feel slightly more “normal” when browsing and using bookmarks.
• The Project Manager has been given a very nice face lift and is far more usable than it was in Beta 7.

Report Writer

We've optimized the layout engine so that reports created with large data sets are now generated SIGNIFICANTLY faster.

Scripting

We've added the following functionality:

• Support for FTP and SFTP to push data back and forth. We're working on some example extensions that will make great use of this functionality.
• A new, improved ListView control.
• A HostPreferences class for storing and retrieving application and extension preferences.
• A HostApp.createPane() and HostApp.createDocument() so that forms can be added to the application as docked panels or documents.

As always, please take the new version for a spin and let us know what you think. Bug reports and all other feedback are more than welcome. Thanks!

You may remember one of our earlier quests on this blog related to tagging the world's public data. We're still eagerly adding and taggin publicly available data sets when we find them.

Today, a friend of mine alerted me to another lode of mine-able data offered by Peter Skomoroch of Data Wrangling. There are a bunch of great sets here; hope you find something to your liking as well.

Besides enjoying his great blog name, I was also happy to be directed to a site called theinfo.org, which simply states it is “for people with large data sets.” It's a site built to gather data-enjoyers across the web and collaborate in three areas:

• Get: scrapers, crawlers, phone calls, buyouts
• Process: conversions, queries, regressions, collaborative filtering
• View: tables, graphs, maps, websites

Here's a quick peek into their mission:

Some of us have spent years scraping news sites. Others have spent them downloading government data. Others have spent them grabbing catalog records for books.Referencement Google And each time, in each community, we reinvent the same things over and over again: scripts for doing crawls and notifying us when things are wrong, parsers for converting the data to RDF and XML, visualizers for plotting it on graphs and charts.

It's time to start sharing our knowledge and our tools. But more than that, it's time for us to start building a bigger picture together. To write robust crawl harnesses that deal gracefully with errors and notify us when a regexp breaks. To start converting things into common formats and making links between data sets. To build visualizers that will plot numbers on graphs or points on maps, no matter what the source of the input.

We've all been helping to build a Web of data for years now. It's time we acknowledge that and start doing it together.

If you love data, this appears to be well worth checking out.

Have a great weekend!