I created some data and stored it several times like this:
with open('filename', 'a') as f: pickle.dump(data, f)
Every time the size of file increased, but when I open file
with open('filename', 'rb') as f: x = pickle.load(f)
I can see only data from the last time.
How can I correctly read file?
Here is Solutions:
We have many solutions to this problem, But we recommend you to use the first solution because it is tested & true solution that will 100% work for you.
Pickle serializes a single object at a time, and reads back a single object –
the pickled data is recorded in sequence on the file.
If you simply do
pickle.load you should be reading the first object serialized into the file (not the last one as you’ve written).
After unserializing the first object, the file-pointer is at the beggining
of the next object – if you simply call
pickle.load again, it will read that next object – do that until the end of the file.
objects =  with (open("myfile", "rb")) as openfile: while True: try: objects.append(pickle.load(openfile)) except EOFError: break
There is a read_pickle function as part of pandas 0.22+
import pandas as pd obj = pd.read_pickle(r'filepath')
The following is an example of how you might write and read a pickle file. Note that if you keep appending pickle data to the file, you will need to continue reading from the file until you find what you want or an exception is generated by reaching the end of the file. That is what the last function does.
import os import pickle PICKLE_FILE = 'pickle.dat' def main(): # append data to the pickle file add_to_pickle(PICKLE_FILE, 123) add_to_pickle(PICKLE_FILE, 'Hello') add_to_pickle(PICKLE_FILE, None) add_to_pickle(PICKLE_FILE, b'World') add_to_pickle(PICKLE_FILE, 456.789) # load & show all stored objects for item in read_from_pickle(PICKLE_FILE): print(repr(item)) os.remove(PICKLE_FILE) def add_to_pickle(path, item): with open(path, 'ab') as file: pickle.dump(item, file, pickle.HIGHEST_PROTOCOL) def read_from_pickle(path): with open(path, 'rb') as file: try: while True: yield pickle.load(file) except EOFError: pass if __name__ == '__main__': main()
I developed a software tool that opens (most) Pickle files directly in your browser (nothing is transferred so it’s 100% private):
Now it’s hosted here: https://fire-6dcaa-273213.web.app/
Edit: Available here if you want to host it somewhere: https://github.com/ch-hristov/Pickle-viewer
Feel free to host this somewhere.
Note: Use and implement solution 1 because this method fully tested our system.
Thank you 🙂