Solved: ValueError: could not convert string to float: ‘’

Python and Pandas

When using Python and Pandas to load and manipulate data it is almost certain that at some point in time you will have to convert data to make it usable to the wider project.

When loading data, in our example originating from a csv file, into a pandas dataframe you will have the data represented as an object type. Even though initially usable there is a common need to transform this into a more usable format, for example a float64 format.

Two options to transform from object to float64 are outlined below

pandas.DataFrame.astype

This is used to cast a pandas object to a specified dtype. In our example we try to transform this into a float64 as show in the example below where we try to transform “avg_use” into a float.

df['avg_use']=df['avg_use'].astype(float)

In general this method works quite well, the exception to this is that you might run into a ValueError as shown below. This commonly is the case if your dataset is not clean and “avg_use” contains values that are not able to be converted to a float. Even though a dataset should be clean, obviously, reality shows that datasets are not always as clean as we would hope for.

ValueError: could not convert string to float: ‘’

Unclean datasets are the main reason that, within dat science projects, a large part of time is spend on collecting, cleaning and preparing the data before the actual use.

pandas.to_numeric

another option available is using pandas.to_numeric which can also be used to convert to a float value as shown in the example below.

using pandas.to_numeric will convert the data to a numeric type. The default return dtype is float64 or int64 depending on the data supplied. Use the downcast parameter to obtain other dtypes.

Please note that precision loss may occur if really large numbers are passed in. Due to the internal limitations of ndarray, if numbers smaller than -9223372036854775808 (np.iinfo(np.int64).min) or larger than 18446744073709551615 (np.iinfo(np.uint64).max) are passed in, it is very likely they will be converted to float so that they can stored in…