GAE – Reading a big CSV blobstore in python

Last week I was trying to upload and parse a big CSV blobstore to Google App Engine (GAE) using python. If you are familiar with GAE you will probably know already that uploading files is not allowed in the standard way, as the application is running in a sandbox within a read-only system. Therefore, if you need to upload something, you have to use the blobstore API provided by google.

In order to do that you need to do the following steps:

1. Request an upload url (somewhere in your app):

from google.appengine.ext import blobstore
url = blobstore.create_upload_url('/destination_after_upload')

This is the url where you have to submit the form with the file and (optionally) any additional data. Once the submission of the form has been completed, the request will be automatically redirected to the provided handler/url (in the example: /destination_after_upload) with all the form data plus the blobstore information.

Note: Everywhere I’ve seen, people assume that you put the generated url in the action of a form in your template/view. However, it is possible for instance to return the url, and then perform a POST request to it programatically, and it will work as well. Example:

#The result of this request will be just the url generated by the previous sample code
upload_url=`curl http://your.appengine.site/get_upload_url/`
#We can curl-post a file to that url, and it will work exactly the same as if we were submitting a form through the browser.
curl -vX POST -F "file=@/path/to/your/file.csv" $upload_url

2. Retrieve the data (in the destination url), for this I’ve used the helper get_uploads:

def destination_after_upload(request):
    # Note that 'file' is the name of the file element used in the form
    upload_files = get_uploads(request, field_name='file', populate_post=True)
    blob = upload_files[0]

3. Parse the CSV file: For this I tried reading the file normally with the csv module, but after facing some issues with the new line characters I ended up using the class BlobIterator and it was working like a charm. Then, I started noticing some random error in the logs due some line of the CSV being corrupted. I was using the following code to parse the CSV:

    # Read CSV content from Blobstore
    blob_reader = blobstore.BlobReader(blob.key())
    blob_iterator = BlobIterator(blob_reader)
    reader = csv.reader(blob_iterator, skipinitialspace=True, delimiter=',')

    # headers = next(reader)
    for row in reader:
        print row
        # do something...
    blobstore.delete(blob.key())

It was all working fine except for some random lines, but I checked the original CSV file and the lines were correct there. I started debugging the execution with PyCharm and I noticed that the issue seemed to be in the next method of the class BlobIterator. Let’s have a look at the code:

def next(self):
        if not self.buffer or len(self.lines) == self.line_num:
            self.buffer = self.blob_reader.read(1048576) # 1MB buffer
            self.lines = self.buffer.splitlines()
            self.line_num = 0

            # Handle special case where our block just happens to end on a new line
            if self.buffer[-1:] == "\n" or self.buffer[-1:] == "\r":
                self.lines.append("")

        if not self.buffer:
            raise StopIteration

        if self.line_num == 0 and len(self.last_line) > 0:
            result = self.last_line + self.lines[self.line_num] + "\n"
        else:
            result = self.lines[self.line_num] + "\n"

        self.last_line = self.lines[self.line_num]
        self.line_num += 1

        return result

I placed a break point in side the first “if” statement, which is the point where it loads one megabyte of data into an internal buffer. The way it works is, summarizing: Load one megabyte of data, split it by the newlines and keep a counter of the current line, returning it on each “next” iteration. The problem is that by loading one megabyte of data, it’s very likely that the last line will be cut in the middle, so you have to concatenate the last line of the first buffer with the first line of the next one, in order to return the proper result. However, the code was already doing it, and the debugger was showing the proper value on the “result” variable, but then when the execution was going back to my “for” loop, the value of the row was actually like this: “last,part,of,first,buffer\n,last,part,of,first,buffer,first,part,next,buffer”

Apparently, it was returning the proper line but prepended by the residual content of the first buffer plus a \n. I was going crazy trying to find out the problem, I even thought that there was an issue on the csv reader module, until I tried to reproduce the issue in a simpler sample:

import unittest
import csv

class TestIterator:
    def __init__(self, data):
        self.data = data
        self.last_line = ""
        self.line_num = 0
        self.lines = []
        self.buffer = None

    def __iter__(self):
        return self

    def next(self):
        if not self.buffer or len(self.lines) == self.line_num:
            self.buffer = self.data[:15]
            self.data = self.data[15:]
            self.lines = self.buffer.splitlines()
            self.line_num = 0

            # Handle special case where our block just happens to end on a new line
            if self.buffer[-1:] == "\n" or self.buffer[-1:] == "\r":
                self.lines.append("")

        if not self.buffer:
            raise StopIteration

        if self.line_num == 0 and len(self.last_line) > 0:
            result = self.last_line + self.lines[self.line_num] + "\n"
        else:
            result = self.lines[self.line_num] + "\n"

        self.last_line = self.lines[self.line_num]
        self.line_num += 1

        return result

class IteratorTest(unittest.TestCase):

    def setUp(self):
        # Read CSV content from string data
        data = """1,2,3,4,5
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5
1,2,3,4,5
"""
        self.test_iterator = TestIterator(data)

    def test_iterator(self):
        reader = csv.reader(self.test_iterator, skipinitialspace=True, delimiter=',')
        for row in reader:
            print row
            if row:
                self.assertEqual(row, ['1', '2', '3', '4', '5'])

def main():
    unittest.main()

if __name__ == '__main__':
    main()

And with this code I was able to reproduce the issue. It turns out that the problem is in the line_num counter that starts from zero and hence it was loading the next buffer one iteration later than it should. I managed to fix it with the following changes:

    def next(self):
        if not self.buffer or len(self.lines) == (self.line_num + 1):
            if self.lines:
                self.last_line = self.lines[self.line_num]
            self.buffer = self.data[:15]
            self.data = self.data[15:]
            self.lines = self.buffer.splitlines()
            self.line_num = 0
            print self.lines

            # Handle special case where our block just happens to end on a new line
            if self.buffer[-1:] == "\n" or self.buffer[-1:] == "\r":
                self.lines.append("")

        if not self.buffer:
            raise StopIteration

        if self.line_num == 0 and len(self.last_line) > 0:
            print 'fixing'
            result = self.last_line + self.lines[self.line_num] + "\n"
        else:
            print self.line_num
            print len(self.lines)
            result = self.lines[self.line_num] + "\n"

        self.line_num += 1

        return result

Note that the first three lines are different and the line “self.last_line = self.lines[self.line_num]” has been moved inside the if. With this changes the iterator seems to work fine when changing between buffer iterations, so the fixed/final version of the BlobstoreIterator is:

class BlobIterator:
    """Because the python csv module doesn't like strange newline chars and
    the google blob reader cannot be told to open in universal mode, then
    we need to read blocks of the blob and 'fix' the newlines as we go.
    Fixed the problem with the corrupted lines when fetching new data into the buffer."""

    def __init__(self, blob_reader):
        self.blob_reader = blob_reader
        self.last_line = ""
        self.line_num = 0
        self.lines = []
        self.buffer = None

    def __iter__(self):
        return self

    def next(self):
        if not self.buffer or len(self.lines) == self.line_num + 1:
            if self.lines:
                self.last_line = self.lines[self.line_num]
            self.buffer = self.blob_reader.read(1048576) # 1MB buffer
            self.lines = self.buffer.splitlines()
            self.line_num = 0

            # Handle special case where our block just happens to end on a new line
            if self.buffer[-1:] == "\n" or self.buffer[-1:] == "\r":
                self.lines.append("")

        if not self.buffer:
            raise StopIteration

        if self.line_num == 0 and len(self.last_line) > 0:
            result = self.last_line + self.lines[self.line_num] + "\n"
        else:
            result = self.lines[self.line_num] + "\n"

        self.line_num += 1
        return result

I hope it helps you to save a headache if you are having the same issue.

How to put image in Navigation Drawer

http://thedeveloperworldisyours.com/wp-content/uploads/putimagen.pngWhen you put image in your navigation drawer you have to declare a layout, in this case relativeLayout.
In layout xml you declare a layout, now you put image inside layout, then you put listView.

<android.support.v4.widget.DrawerLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/drawer_layout"
    android:layout_width="match_parent"
    android:layout_height="match_parent" >

    <!-- Contenido Principal -->

    <FrameLayout
        android:id="@+id/content_frame"
        android:layout_width="match_parent"
        android:layout_height="match_parent" >
    </FrameLayout>

    <!-- Menú Lateral -->

    <RelativeLayout
        android:id="@+id/left_drawer"
        android:layout_width="240dp"
        android:layout_height="match_parent"
        android:layout_gravity="start"
        android:background="@drawable/tabbar"
        android:orientation="vertical" >

        <ImageView
            android:id="@+id/image_view"
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:background="@drawable/image_drawer_list" />

        <ListView
            android:id="@+id/list_view_drawer"
            android:layout_width="fill_parent"
            android:layout_height="fill_parent"
            android:layout_below="@id/image_view"
            android:choiceMode="singleChoice" />
    </RelativeLayout>

</android.support.v4.widget.DrawerLayout>

In your class you declare

private RelativeLayout mDrawerRelativeLayout;
package com.example.navegationdrawer;

import android.content.res.Configuration;
import android.os.Bundle;
import android.support.v4.app.ActionBarDrawerToggle;
import android.support.v4.app.ActivityCompat;
import android.support.v4.app.Fragment;
import android.support.v4.app.FragmentManager;
import android.support.v4.widget.DrawerLayout;
import android.support.v7.app.ActionBarActivity;
import android.view.Menu;
import android.view.MenuItem;
import android.view.View;
import android.widget.AdapterView;
import android.widget.AdapterView.OnItemClickListener;
import android.widget.ArrayAdapter;
import android.widget.ListView;
import android.widget.RelativeLayout;
import android.widget.Toast;

import com.example.navegationdrawer.fragment.FirstFragment;
import com.example.navegationdrawer.fragment.SecondFragment;
import com.example.navegationdrawer.fragment.ThirdFragment;

public class MainActivity extends ActionBarActivity {

	private String[] mOptionMenu;
	private DrawerLayout mDrawerLayout;
	private RelativeLayout mDrawerRelativeLayout;
	private ListView mDrawerList;
	private ActionBarDrawerToggle mDrawerToggle;

	private CharSequence mTitleSection;
	private CharSequence mTitleApp;

	@Override
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.activity_main);

		mOptionMenu = new String[] { "Opción 1", "Opción 2", "Opción 3" };
		 mDrawerLayout = (DrawerLayout) findViewById(R.id.drawer_layout);
		 mDrawerRelativeLayout = (RelativeLayout)
		 findViewById(R.id.left_drawer);
		 mDrawerList = (ListView) findViewById(R.id.list_view_drawer);
		 mDrawerList.setAdapter(new ArrayAdapter<String>(getSupportActionBar()
		 .getThemedContext(), android.R.layout.simple_list_item_1,
		 mOptionMenu));

		 mDrawerList.setOnItemClickListener(new OnItemClickListener() {
		 @Override
		 public void onItemClick(AdapterView<?> parent, View view,
		 int position, long id) {

		 Fragment fragment = null;

		 switch (position) {
		 case 0:
		 fragment = new FirstFragment();
		 break;
		 case 1:
		 fragment = new SecondFragment();
		 break;
		 case 2:
		 fragment = new ThirdFragment();
		 break;
		 }

		 FragmentManager fragmentManager = getSupportFragmentManager();

		 fragmentManager.beginTransaction()
		 .replace(R.id.content_frame, fragment).commit();

		 mDrawerList.setItemChecked(position, true);

		 mTitleSection = mOptionMenu[position];
		 getSupportActionBar().setTitle(mTitleSection);

		 mDrawerLayout.closeDrawer(mDrawerRelativeLayout);
		 }
		 });
		 mDrawerList.setItemChecked(0, true);
		 mTitleSection = getTitle();
		 mTitleApp = getTitle();

		 mDrawerToggle = new ActionBarDrawerToggle(this, mDrawerLayout,
		 R.drawable.ic_navigation_drawer, R.string.drawer_open,
		 R.string.drawer_close) {

		 public void onDrawerClosed(View view) {
		 getSupportActionBar().setTitle(mTitleSection);
		 ActivityCompat.invalidateOptionsMenu(MainActivity.this);
		 }

		 public void onDrawerOpened(View drawerView) {
		 getSupportActionBar().setTitle(mTitleSection);
		 ActivityCompat.invalidateOptionsMenu(MainActivity.this);
		 }
		 };

		 mDrawerLayout.setDrawerListener(mDrawerToggle);

		 getSupportActionBar().setDisplayHomeAsUpEnabled(true);
		 getSupportActionBar().setHomeButtonEnabled(true);
	}

	@Override
	public boolean onCreateOptionsMenu(Menu menu) {

		getMenuInflater().inflate(R.menu.main, menu);
		return true;
	}

	@Override
	public boolean onOptionsItemSelected(MenuItem item) {

		if (mDrawerToggle.onOptionsItemSelected(item)) {
			return true;
		}

		switch (item.getItemId()) {
		case R.id.action_settings:
			Toast.makeText(this, "Settings", Toast.LENGTH_SHORT).show();
			;
			break;
		default:
			return super.onOptionsItemSelected(item);
		}

		return true;
	}

	@Override
	protected void onPostCreate(Bundle savedInstanceState) {
		super.onPostCreate(savedInstanceState);
		mDrawerToggle.syncState();
	}

	@Override
	public void onConfigurationChanged(Configuration newConfig) {
		super.onConfigurationChanged(newConfig);
		mDrawerToggle.onConfigurationChanged(newConfig);
	}
}

Multiple OnClickListener android

You can do the onClick() method one time.

First step

implements OnClickListener

Then you have a error
add unimplemented method

 @Override
	public void onClick(View v) {
		// TODO Auto-generated method stub
        }

This is full code:

public class OneFragment extends Fragment implements OnClickListener{

	private Button mButtonOne;
	private Button mButtonTwo;
	private Button mButtonThree;
	private static View mView;

	@Override
	public View onCreateView(LayoutInflater inflater, ViewGroup container,
			Bundle savedInstanceState) {
                if (mView == null) {
                    mView = inflater.inflate(R.layout.fragment_one, container, false);
	        }

                mButtonOne = (Button) mView.findViewById(R.id.one_button);
		mButtonTwo = (Button) mView.findViewById(R.id.two_button);
		mButtonThree = (Button) mView.findViewById(R.id.three_button);

		mButtonOne.setOnClickListener(this);
		mButtonTwo.setOnClickListener(this);
		mButtonThree.setOnClickListener(this);

                return mView;
	}
        @Override
	public void onClick(View v) {
		// TODO Auto-generated method stub
		if (v==mButtonOne) {
			changeSecction(SECTION_ONE);
		}else if (v==mButtonTwo) {
			changeSecction(SECTION_TWO);
		}else {
			changeSecction(SECTION_THREE);
		}
	}

How to get coordinates of an address in android

public class MapActivity extends android.support.v4.app.FragmentActivity{

	private static final int MAP_ZOOM = 15;
	private String mAddress;
	private GoogleMap mMap;
	private double mLongitude;
	private double mlatitude;

	@Override
	protected void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.activity_map_hotel_car);

		Bundle extras = getIntent().getExtras();
		if (extras != null) {
			mAddress = extras.getString("Address");
		}
		mMap = ((SupportMapFragment) getSupportFragmentManager()
				.findFragmentById(R.id.map)).getMap();

		Log.v("ddd", mAddress);

		Geocoder geocoder = new Geocoder(this, Locale.getDefault());
                //To initialice list of address
		List<Address> addresses = null;
		try {
                        //To put the address in list
			addresses = geocoder.getFromLocationName(mAddress, 1);
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
                //Get Latitude and longitude
		Address address = addresses.get(0);
		mLongitude = address.getLongitude();
		mlatitude = address.getLatitude();
		showMarker(mlatitude, mLongitude);
		centerMapOnMyLocation();

	}

	//show the poi in map
	private void showMarker(double lat, double lng)
	{
	    mMap.addMarker(new MarkerOptions()
	        .position(new LatLng(mlatitude, mLongitude))
	        .title("Pais: España"));
	}
	//Center the point in map
	private void centerMapOnMyLocation() {
			mMap.moveCamera(CameraUpdateFactory.newLatLng(getMyLatLng()));
			mMap.animateCamera(CameraUpdateFactory.zoomTo(MAP_ZOOM));
	}
	private LatLng getMyLatLng() {
		return new LatLng(mlatitude, mLongitude);
	}
}

Go to google maps app with a address

The first step, you put the packaget for google maps

private String PACKAGET_MAPS="com.google.android.apps.maps";

The seconds step, you put query for to search the address.

private String HTTP_MAPS = "http://maps.google.com/maps?q=";

The third step, you put intent

if (Utils.isPackageInstalled(PACKAGET_MAPS, getApplicationContext())) {
StringBuilder stringBuilder = new StringBuilder();
//query of google maps
stringBuilder.append(HTTP_MAPS);
//String with your address
stringBuilder.append(separtorWithPlus("yourAddress"));
stringBuilder.append(",+");
//string of City
stringBuilder.append((separtorWithPlus("City")));
//intent with Uri which has query with address and city
intent = new Intent(android.content.Intent.ACTION_VIEW,
Uri.parse(stringBuilder.toString()));
startActivity(intent);
} else {
mShowMessage.showAlert(getResources().getString(
R.string.string_google_maps_uninstall));
}