DataWeave

2012 - The year that was

Tue, 01 Jan 2013 08:56:00 +0530

We at DataWeave had a fast-paced 2012 with a large, at times, mind-bogglingly so ,number of threads running in parallel. As we draw close to the New Year, we want to pause a moment and look back at the year that was.

We launched DataWeave and PriceWeave in February 2012.

The number of data points that we served as on 28th December is 44,64,310.

This approximates to about 14,000 data points every day on average, or about 10 every minute. We are proud of these stats, but we are not ones to be complacent. We instead want to keep pushing at those numbers constantly, and keep treading deeper waters.

Customers

Hoopos, Rediff, Indiatimes Shopping, CrowdAnalytix, Snapdeal, Puma, UrbanJourney, Jabong, OfficeYes, EasyRation, Zigwheels are some of our esteemed customers.

Applications

CostShaker, Lowprice, Gaadiwala, Localbazaar.

Customer Segments

Comparison Shopping Engines, eCommerce Portals, Brands, Developers, Travel Companies and Media Houses.

Mobile Apps

While most of our initial focus has been on developing offerings for businesses, we also want to provide an active space for the developer community. There is a clear need for mobile apps powered by high quality datasets, which we aim to fulfill. A couple of Apps have already been powered by us, lending teeth to our capabilities.

LowPrice was the first Android app that is powered by our commodity prices datasets. It lets you scan a book and find the lowest priced seller. Pranay Airan, who developed this, shares his thoughts here

GaadiWala was developed by Vamsi Krishna to keep users informed about the car and bike prices across cities in India.

Events

We graced every event where DataWeave’s presence was needed:

HackNight, The Fifth Elephant, Pycon, Droidcon, Techsparks 2012, Nasscom Product Conclave (2012) and GSFIndia.

Achievements

We were named as one of the 3 best Tech Product Startups in India for 2012 by TechSparks.

Media Coverage

The media coverage we have received this year has been heartening and validates much of our very own beliefs. Some of the write ups about us are on TechCircle, and we were featured twice on YourStory and once on CNBC

http://techcircle.vccircle.com/500/tlabs-startup-accelerator-report-card-1-dead-2-up-for-funding-and-4-evolving/

http://yourstory.in/2012/04/do-you-find-data-on-the-web-too-cumbersome-dataweave-in-simplifiesthe-problem/

http://yourstory.in/2012/09/techsparks-2012-unveils-the-top-30-tech-product-startups-from-india/

We are growing …

We added two awesome team members towards the end of the year. Sachin and Abhishek

We would like to thank everybody at Headstart for their support and organizational competence and in helping us secure two bright talents.

What to look for in 2013 …

Lot more datasets and APIs
More verticals—travel, corporate intelligence
Additional geographies
Increased focus on developers and mobile app space
Customized Alerts

More people coming on board

A few more exciting stuff … Some of which we’ll only be able to let you in on officially at the beginning of 2013!

Thank You!

Tlabs – for bootstrapping and continuing to support us. A shout out to Abhishek and Pankaj (now at 500 startups)!

Thanks a lot to our advisors: Gautam Sinha and Miten Sampat.

Special thanks to the numerous people who gave us time to meet them and discuss our ideas.

Happy New Year! Let us look forward to an exciting 2013!

How we created Lowprice Android app - Part 2

Mon, 03 Dec 2012 08:00:00 +0530

In part-1 of this post, I discussed about the BookSearch API provided by DataWeave. In this part, I will discuss the Scan & Search functionality and integrating these to complete our Android App.

SCAN:

The ISBN number of a book can be obtained by scanning the bar code directly using the camera on your android phone. LowPrice, by itself is not bundled with this capability. To achieve this functionality, we use a native concept on Android called Intents. Intents allows us to delegate required functionality to already existing apps, thus eliminating the need to reinvent the wheel. Google provides an open source barcode scanner app ZXING. We will now use Intents to delegate the barcode scanning functionality to ZXING.

 Intent intent = new Intent("com.google.zxing.client.android.SCAN");
 intent.putExtra("SCAN_MODE", "QR_CODE_MODE"); 
 startActivityForResult(intent, 0);

We create an intent to use the barcode scanning functionality of ZXING in the first line. Then we define the mode in which the scanning has to be started. Different modes for different uses are defined on the ZXING website. Once the intent is created, we now start the intent and listen for results from the barcode scanner. Starting an intent will launch the corresponding application.

To get data from the barcode scanner:

 public void onActivityResult(int requestCode, int resultCode, Intent intent) { 
 if (requestCode == 0) { 
 if (resultCode == RESULT_OK) {
 String contents = intent.getStringExtra("SCAN_RESULT"); 
 String format = intent.getStringExtra("SCAN_RESULT_FORMAT");

Once ZXING detects a valid barcode, the function defined above will be invoked, which will read the barcode and convert it to string for compatibility with the BookSearch API. This simple piece of code will provide our app with all the functionalities of a barcode scanner.

Integrating both

Now that we have the ISBN number from the Scan module, we can use the searchByIsbn method from DataWeave’s BookSearch API to fetch the price of the book across various vendors in India.This is described in detail in part-1 of this post.

This is the story of LowPrice for now. Creating an Android App made easy. Do expect more in the future.

I can be reached at pranay.airan@iiitb.net . Feel free to provide your feedback at contact@dataweave.in .

Download lowprice android app now https://play.google.com/store/apps/details?id=com.binarybricks.lowprice

How we created Lowprice Android app

Mon, 19 Nov 2012 19:20:00 +0530

[Guest post from Pranay Airan who’s an early adopter of DataWeave. Pranay is enthusiastic about app development. Below he shares his experiences of developing LowPrice, an Android app powered by DataWeave’s pricing API.]

In this two part blog post, I will discuss how I created LowPrice using DataWeave’s APIs and a custom backend. LowPrice (https://play.google.com/store/apps/details?id=com.binarybricks.lowprice) is an Android app that gets you the lowest price for books across 9 online stores in India simply by scanning the book’s barcode. LowPrice also lets you search books by title, author, or publisher, so you don’t even have to go to a store really!

In Part 1, let’s look at the relevant APIs provided by DataWeave, understand how to query data, and how to consume this data through an Android app. In Part 2, we will see how I created the Scan and Search features and how you can achieve the same. Finally, we will wire the code from these two posts to create a complete app.

DataWeave API and JSON

DataWeave provides APIs around public data available on the web. One of their APIs is around providing prices of books across various ecommerce stores in India. LowPrice uses this API http://www.dataweave.in/apis/dataset-Book-Price-Search-By-ISBN—19.html. This API covers a lot of different sources including some of the top ecommerce stores in India like Landmark, Infibeam, and HomeShop18.

For an exclusive access to their APIs, register on http://dataweave.in/ and get an access key. This gives you access to all the data APIs provided by DataWeave.

Lets look at the API for book prices:

1. Search by ISBN

This feature enables you to query for the price of a book across various stores by ISBN number of the book. The API call for the same is http://api.dataweave.in/v1/book_search/searchByIsbn/?isbn=9788174369109 . The API returns data as JSON. The response format is as follows:

{
source: "Landmark",
url:"http://www.landmarkonthenet.com/beyond-the-lines-an-autobiography-by-kuldip-nayar-books-9788174369109-22248864/",
listprice: "595",
price:"399",
thumbnail: "static.landmarkonthenet.com/9788174369109/m800/m800/?dept=books&shot=0",
publisher: "Roli Books Pvt Ltd",
author: "Kuldip Nayar"
}

This response contains the URL of the book, source, MRP of the book, price at which the book is being sold, publisher, and author. The Scan and Search feature of LowPrice uses this API to get the prices of books. How does the App get the ISBN number? That will be explored in Part 2.

2. Search by Title/Author/Publisher

Not always does everybody have the ISBN number of a book that they want. Most of the times, we know the title/author/publisher of a certain book. To enable free text search, the API also provides a method to search by these attributes of a book as well. LowPrice uses this API to power book search. If we would like to search by the name of a book eg: Harry Potter,
http://api.dataweave.in/v1/book_search/searchByTitle/?title=harry%20potter . The response from the API is as follows:

 {
 title: "HARRY POTTER AND THE CHAMBER OF SECRETS",
 isbn: "9781408810552",
 author: " J. K. Rowling",
 mrp: " 399",
 available_price: "263",
 thumbnail: "http://img8.flixcart.com/image/book/5/5/2/harry-potter-and-the-chamber-of-secrets-100x100-imad8qthkqxzw5q5.jpeg",
 publisher: "BLOOMSBURY"
 }

Now that we have the data in an easy to consume format, let us now focus on building the Android app itself.

How to do a REST call in Android

All of DataWeave’s API are REST based. Let us see how we can consume a REST API in Android.

 DefaultHttpClient httpClient = new DefaultHttpClient();
 HttpGet httpGet = new HttpGet("http://api.dataweave.in/v1/book_search/searchByTitle/?title="+bookname);
 HttpResponse httpResponse = httpClient.execute(httpGet);
 HttpEntity httpEntity = httpResponse.getEntity();

With this simple 4 lines of Code we are able to consume the data in our app. Since the API returns the data as JSON, lets see how to parse this data.

How to parse JSON in Android

Android SDK provides us with an option to natively parse JSON. Optionally we could also use Jackson or GSON.

 JSONArray json = null;
 List mElementList = new ArrayList();
 json = new JSONArray(jsonString);

This simple code will help you to parse JSON response which we receive from our REST API.

Summary

In this post we took a brief look at DataWeave’s API which is powering LowPrice backend and we also saw how to consume this API in android and use the data in a way we want. In the next post, I will explain how I built the Scan and Search feature in LowPrice.

Building data apps has never been easier!

Feel free to download and check out lowprice android app from here: https://play.google.com/store/apps/details?id=com.binarybricks.lowprice

Please share your feedback.

Story of a Data API

Tue, 16 Oct 2012 15:45:00 +0530

Data is ubiquitous on the web. Data is available to anybody who wants to make use of it. Businesses and developers can greatly benefit from data. All the above are given. But the problem is that data is spread across a large number of sources; it is not clean; there exists duplicate data; and it keeps getting updated all the time. There is no one solution that exists today, that can make this data consumable without going through hoops around crawling, cleansing and then consuming data per source.

DataWeave is all about providing easy access to high quality datasets. The primary mode of access to our datasets is through our Data APIs. In this post, we’d like to share the story of a data API: our process from understanding a widespread need—often latent—for a certain kind of data to delivering the data through APIs. In brief, this process involves the following steps:

identifying/understanding a need (often latent) for data
identifying the right data sources
crawling/scraping the relevant information
pre-processing/storing/post-processing
data cleaning/deduplication
developing/publishing the API
customizing the API based on specific requests

3,4,5 are ones that we need to repeat periodically based on the data update rates of the sources as well as consuming applications’ needs for freshness of data. 2 and 7 are the ones in which we customize our offerings based on specific customer requests.

Below the fold, we illustrate the process through one of our popular datasets, Commodity Prices dataset.

What is Commodity Prices API

The commodity prices dataset comprises daily arrival and prices data of different agricultural commodities as received from the Agricultural Produce Market Committee (APMCs) of different states in India. Each of these APMCs publishes information independently.The structure of a sample commodity record represented as a JSON object looks like this:

"data": [ { "date": "2011-12-02", "commodity": "Ajwan", "state": "Andhra Pradesh", "market": "Kurnool", "Arrivals_Tonnes": "3.9", "origin": "", "variety": "Other", "Minimum_Price": "8590", "Maximum_Price": "13069", "Modal_Price": "10800", "unit": "Rs./Quintal" } ],

The API provides access to the above datasets in multiple ways, such as: list of all commodities and find volume and prices by commodity.

How we built Commodity Prices API

Identifying Need for Data

Commodity prices are needed by farmers who want to sell their produce; by wholesale dealers who want to stock up their supply; and by market analysts who want to analyze and predict trends. This data can be an end in itself or it might be the prerequisite for something more. But whatever the need, the amount of effort expended in getting this data is enormous. We know for a fact that so far this has involved dispatching someone to get the list of prices from the APMC, or cajoling the personnel there to call over the phone at the close of a day.

Well, the commodity prices data is published online by the respective state agricultural and marketing boards as well as the national agricultural market association! So, what is required simply is an automated way of extracting and organizing this data. However, the way in which this data is published varies across all these sites. The difference can be observed right from name of a particular field to the unit of representation. When trying to collate data from these sources, the amount of effort demanded of the developer/journalist/user is quite high. The end might not seem so bright in comparison to this arduous journey!

Identifying Data Sources

Source identification is one of the biggest challenges we face every day. How to determine the validity of a source? When will a source publish? How often do we have to poll the source to check for information? How do we ensure we are not taxing the resources of our sources? Is it possible that something that we did in good faith turns out to be unethical inadvertently? These are some questions that we need to answer. Even if we know which government department is responsible for publishing the data we want, the department itself is spread across multiple websites each holding part of the information we need to put together. Or it might turn out that the same data is published on all these sites, but the rate at which the data is published varies. Some sources might stop publishing data without any notifications while we are still waiting for a fresh set of data points to be populated. Identifying sources for a data API is not a one time effort. It requires us to keep ourselves abreast of all the alternative sources that might publish information.

Crawling and Scraping

It appears once we have identified the data sources and understood (with some pain, you admit!) the various formats and patterns in which they publish data, the process of crawling these sources is very simple. Well, not really! There awaits a goblin at the turn of the street. One thing about web programming is that there is no single standard to write a website in (plain old PHP/.NET/JSP/Ajax based). This is a boon as well as a bane. Its a boon to web developers. They might find it easier to develop the website in a certain framework. But its a bane for people trying to use this data in a machine readable format. There is no one size fits all solution for this. You will have to understand the functioning of each of these frameworks. Sometimes, websites implement a combination of techniques to make navigation/usage better. Trying to generalize solutions based on these factors is quite difficult.

Pre-processing/storing/post-processing

Once the raw data is available with us, we need to figure out an appropriate schema for storage. Is the data clean enough to be consumed or is there a data sanitization phase? This will determine the additional steps before the data API can be published. The data sanitization phase could entail something as trivial as formatting the date properly to something as complex as breaking a phrase into constituent terms, identify metadata for these terms and normalize the metadata. The data thus accumulated should be stored efficiently to be served effectively. Some of the datasets might be more suited for a traditional RDBMS storage while others might be suited for an alternative storage models. If the data has to be served from an RDBMS, a careful decision has to be taken whether to de-normalize the data or store it as is.

Data cleaning/deduplication

As mentioned previously, a data API can be powered by multiple sources. There might or might not exist any overlap in the data published by these sources. How to deduplicate data once we have crawled them is a separate problem of its own might. The deduplication process could mean just deleting duplicate records from the database. But the way a source publishes data could be very different. For example, the price of a certain commodity may be published by multiple sources for the same market/location depending on whose jurisdiction the market/location might lie in. They might not represent the same location in the same way. E.g., Bangalore, Bengaluru or Bengalooru. Identifying these as variants of the same word or concept is a non-trivial problem.

Developing and publishing API

Publishing an API requires us to understand how a developer/ user might use the API. We cannot always expect a user to download the complete dataset and run these aggregates on their end. The user might not always be executing his application on a beefy computer with all the requisite software packages.

This requires us to look at various possible views for each dataset, come up with methods that would meet these needs.

Customizing API for specific requests

As discussed above, there might be a large number of views on each dataset. It is not always easy to comprehend or anticipate the consumers’ usage patterns. There might be cases where a particular business application needs might be difficult to cater to from the APIs available with us. In such cases, our users place requests for specific APIs (pertaining to their access patterns) to be made available on this data. DataWeave does take up and implement such customization requests on a case to case basis.

Please do sign up for an API key and try out our APIs. If you have any suggestions/ ideas/ questions please send it to contact@dataweave.in.

Our worldview of data

Wed, 22 Aug 2012 09:29:53 +0530

We talked about our Data APIs at the startup saturday held recently in Bangalore, India. You can find the slides here. We presented our worldview of data and the unique challenges in dealing with different “kinds” of data.

The image above signifies the two fundamental axes that helps us in classifying data. The horizontal axis signifies temporality while the vertical axis represents the presence or absence of structure underlying the data. Any “kind” of data that we might think of falls into any one of these quadrants.

The reason why we try and classify data into one of these quadrants is because the underlying challenges of dealing with data from any of these quadrants are inherently different. For example, data that is unstructured requires sophisticated and text mining techniques to derive value from the data, while mining data based on freshness becomes important when dealing with data that is temporal in nature.

Most of the datasets that we deal with at DataWeave are primarily unstructured and temporal.

Data driven Journalism and DataWeave

Tue, 15 May 2012 16:20:00 +0530

Wikipedia states: data-driven journalism as a journalistic process based on analyzing and filtering large data sets for the purpose of creating a news story. Data driven journalism enables reporters to tell meaningful stories with insights gleaned from data using various tools.

We are observing a growing trend in which Governments and other institutions are making their data publicly available. In addition, most of this data is available online. This presents a significant opportunity for the journalism industry.

Well, but the fly in the ointment is that this data is not readily consumable because

The data is dispersed across various sources. Sifting through such dispersed data is a complex and time consuming task.
The data is temporally changing. So, it is difficult to keep track of it.
There is no standard representation for this data.

This means that once a potential data source is identified, there is the tedious task of cleaning, structuring and transforming the data to a consumable form. But this is self-defeating. Should a journalist really spend so much time and effort in collating data reports, while her actual goal is to create cohesive news stories? To put it in another way: one needs to be a technical person to derive value out of the huge amount of data available. However, typically it is a non-technical person such as a journalist or a market researcher who can make most sense and draw meaningful conclusions out of that data!

Imagine a situation where generating a data report is just a few clicks away. This is where DataWeave comes in. DataWeave makes the process of accessing data really easy, by aggregating data from various public sources and providing them via easy to access APIs in a consumable form.

Using DataWeave’s Data APIs, a user can selectively query for data and use them for gaining insights. Our APIs can also be easily plugged into the user’s system – in this case, a journalist’s data report generation dashboard. Once the data is available in a standard, consumable format, it is easy to visualize or identify patterns in it using only a few lines of code, or by exporting it to spreadsheet tools.

Let’s consider a scenario in which a journalist wants to analyze the change in retail prices of commodities over time, for a particular city. She can use DataWeave’s Retail Commodities API to access this data. The required data can be queried through the dashboard provided in the link, for instance. Similarly, the API could be plugged into any existing report generation dashboard, for performing analysis.

Taking our own dashboard as an example, let us look at how the prices of potato have varied in Mumbai for the month of March, 2012.

You can then download the data as a csv file which can be imported to your favourite spreadsheet program or visualization kit. It is as easy as that!

A basic analysis of price changes can be visualized using MS Excel graphs, for instance.

Similarly, the API can be plugged into dashboards with rich visualizations and various “key performance indicator” analyses – the journalist’s dashboard, for instance!

Please share your thoughts and experiences with us at contact [AT] dataweave [DOT] in.

DataWeave at Open DataCamp Bangalore

Wed, 18 Apr 2012 00:11:00 +0530

Slides from our presentation at the Open DataCamp held in Bangalore on March 24th, 2012.

Open Data and APIs - DataWeave

View more presentations from DataWeave

What can you do with data APIs?

Tue, 10 Apr 2012 12:43:00 +0530

Adhering to our core philosophy, “make open data on the web easily accessible and consumable”, we have launched a few APIs around open data we have been monitoring for about an year now. How can these APIs help you? APIs reduce the development time of data-dependent apps considerably, as you can focus on more relevant aspects of your application, such as algorithm design, visualizations, application interfaces, etc, than be stuck in the data collection/curation phase.

We realized the power of this as we started consuming from our API firehose to build our first product - PriceWeave. Priceweave is built on top of the ecommerce product pricing API.

What’s so great about “yet another price comparison engine”, might be the first question in your mind. PriceWeave is more than just a price comparison engine. Our goal is to help businesses get access to data required to make informed decisions. These could range from a simple price monitoring service to real time product placement strategies, short term sales forecasting, deciding how best to spend investor dollars (other than bleeding on discounts).

PriceWeave is being used by our customers as a real time competitor monitoring engine. Priceweave can help businesses with:

1. Real time monitoring of prices across competitors.
2. Set alerts for price changes.
3. Easily integrate it with your own Inventory Management Systems to produce a sales forecast report.
4. Use the dashboard to mine through data easily. (By recently changed, By lowest priced competitor etc).

The screenshots here will give you a visual account of how PriceWeave operates.

Monitor your products in real time

Map your products to the products that we already have.

Surf through the data using a set of filters available on the dashboard.

Here is a list of most commonly asked questions about priceweave http://www.priceweave.com/help.php

PriceWeave is just one of the many possibilities that can be built by leveraging our ecommerce API. You can mash it up with the other APIs we provide and come up with interesting apps or visualizations. In fact, we would love to hear about such undertakings! Please share your thoughts and experiences with us @ contact[AT]dataweave[dot]in.

Meet DataWeave ...

Mon, 02 Apr 2012 14:21:00 +0530

A month has passed by since the launch of DataWeave’s new website. We have a seen a steady increase in usage and a few paying customers.

DataWeave is about making data on the Web easily accessible and consumable. Be it in creating applications or gaining insights in data, the first step any user goes through is: “Get the right data in the right form”. Though a lot of data sources exist on the Web, it is hard to find these sources and access them in a standard format. We create data APIs that lets users easily access this data in standard formats.

Some of the APIs that are available are eCommerce pricing data, Commodity prices, Wholesale price index. Check out the list of APIs available here.

We can think of a lot of use-cases that one can come up with using these APIs:

An application to monitor commodity prices have any impact on the share prices of a company. Companies use raw materials and the prices of these raw materials have a certain impact on their sales and hence share prices.
Impact of population growth on weather.
Did the weather have any impact on the number of people who turned up to vote during elections?

We believe the use cases around this data are limitless.

The ability to create data mashups using these APIs and/or combine them with proprietary data opens up a lot of use cases around this data. For example, organizations are using the eCommerce pricing data API to monitor competition, educate vendors and a host of other applications.

Data journalists, market researchers, analysts and application developers will find this data immensely useful.

In the days to come, we plan to add more data APIs and add wrappers along with the documentation around this data to enable easy consumption of this data.

Would love to hear your feedback.

Welcome to DataWeave

Wed, 07 Sep 2011 17:53:43 +0530

DataWeave is a data marketplace that enables data-driven businesses to access, search and visualize vast repositories of curated data from the Web in a form they can derive value from.

Our offerings include:

Curated datasets across various domains
Example datasets: commodity pricing data, government census data, location data, election data, scientific/academic datasets, weather data etc.
Standard as well as customized visualizations over data
Example visualizations: wikipedia entity relationships, trends in census data, trends in real estate pricing, etc.
Customized data tracking services
Example services: monitoring temporally changing data such as global crude oil prices, keyword tracking on social media, tracking and comparing commodity prices across vendors, etc.

We believe that Open Data offers huge opportunities for businesses to add value to their products and services. It also provides businesses new avenues for innovation.

Academic and research organizations can also benefit from Open Data, as it offers them more opportunities for validating their research ideas and proofs of concepts. It also provides them more avenues for conducting research. Similarly, Open Data provides enormous opportunities to NGOs to better understand the problems and needs of the society at large. It helps them draw up plans of action for tackling today’s problems.

In order to achieve the ideal of Open Data, data needs to be easily accessible, available in standard formats, and through simple API on an on-demand basis. We need to liberate data that is hidden in domain dependent or unstructured formats or as archived files. We need to provide unified access to related data spread across many sources.

With this belief we have created DataWeave. We at DataWeave passionately believe in the unreasonable effectiveness of data in businesses. Talk to us! We can help you weave your success stories!