Social Hygiene: Pruning my Twitter Feed with Plumes

While scrolling through my Twitter feed recently, I began to get a little annoyed at the amount of content that I was simply not interested in. In engineering terms, my signal-to-noise ratio was way too low.

This led me to think about my social media hygene and how I don’t tend to prune my tweets, people I follow, or topics of interest.

So, over a few days off, I made a tool: plumes.

I designed plumes to be a simple Twitter CLI for day-to-day social media hygiene, allowing me to perform basic pruning operations. My end goal was to make a cron job that would perform typical operations at a scheduled interval.

In this post, I’ll be walking through my pruning process using plumes as I try to achieve a more reasonable level of Twitter content quality. I have two overall goals:

1. Prune my tweets: delete old and low-value tweets
2. Prune my friends: unfollow people based on certain criteria

Extracting Twitter Data

First, we need data.

The plumes CLI makes this pretty straight forward:

plumes tweets
plumes friends

The above commands give me all my tweets and all my friends (i.e., people I follow) as two JSON files, EngNadeau-tweets.json and EngNadeau-friends.json, respectively.

Analyzing and Pruning Tweets

Let’s take a look at my tweets. Using the JSON output from plumes, we can load the data and get an idea of my tweeting habits and quality.

Loading Data

Loading the data is a simple JSON -> DataFrame process:

import json
from pathlib import Path
import pandas as pd

# nicer pandas float formatting
pd.options.display.float_format = "{:g}".format

# load data
path = Path("EngNadeau-tweets.json")
with open(path) as f:
    data = json.load(f)

# convert to pandas dataframe
df = pd.json_normalize(data).pipe(
    lambda x: x.assign(**{"created_at": pd.to_datetime(x["created_at"])})
)

df

Exploring Data

Wow, I never thought of myself as a “tweeter”, but 1600+ tweets is pretty cool.

Moreover, the data is very nicely structured, and the pandas.normalize_json() is wonderful at converting the semi-structured JSON data into a flat table.

Let’s take a closer look as some statistics:

df[["retweet_count", "favorite_count"]].describe()

Hmmmm, I don’t remember ever going viral and getting 200k+ retweets.

Per the Twitter API docs, the retweet_count key of a Tweet object counts the source tweet’s retweets, not just my personal retweets.

Let’s filter out retweets to get just my personal tweets. Since we’re beginning to chain conditions, filters, transforms, etc. on our dataframe, I’ll begin using pipes to keep the code clean and efficient. (Note: I LOVE pipes).

(
    df.pipe(lambda x: x[~x["retweeted"]])
    .pipe(lambda x: x[["retweet_count", "favorite_count"]])
    .describe(percentiles=[0.5, 0.9, 0.95, 0.99])
)

Aha! That’s more like it.

So I have around 1.3k personal tweets and I definitely never went viral.

Tweets vs. Time

I wonder what this looks like over time?

from matplotlib import pyplot as plt
import matplotlib as mpl

mpl.rcParams["axes.spines.right"] = False
mpl.rcParams["axes.spines.top"] = False

figsize=(10, 4)

fig, ax = plt.subplots(figsize=figsize)

(
    df.pipe(lambda x: x[~x["retweeted"]])
    .set_index("created_at")
    .resample("2Q")
    .count()
    .pipe(lambda x: x.set_index(x.index.date))
    .pipe(lambda x: x["id"])
    .plot.bar(ax=ax, rot=30)
)

fig.suptitle("Tweets Over Time")
ax.set_xlabel("Quarter")
ax.set_ylabel("Number of Tweets")
fig.tight_layout()

It appears that I tend to be cyclical with my tweets. Here are some highlights I can think of:

• 2015-2016: Attended medical conferences and was tweeting to promote my company.
• 2017-2018: Attended robotics conferences and was tweeting to promote my research.
• 2019: Writing my PhD thesis and forgot about the rest of the world.

Interactions vs. Time

fig, ax = plt.subplots(figsize=figsize)

(
    df.pipe(lambda x: x[~x["retweeted"]])
    .set_index("created_at")
    .resample("2Q")
    .sum()
    .pipe(lambda x: x.set_index(x.index.date))
    .pipe(lambda x: x[["retweet_count", "favorite_count"]])
    .plot.bar(ax=ax, rot=30)
)

fig.suptitle("Interactions Over Time")
ax.set_xlabel("Quarter")
ax.set_ylabel("Number of Interactions")
ax.legend()
fig.tight_layout()

As expected, the number of interactions generally follows my tweeting frequency. The more you give, the more you get.

But, I did have a stellar quarter in 2017.

Pruning Tweets

With goal #1 in mind, let’s use plumes to prune (i.e., delete) old tweets that aren’t worth keeping around. Judging from the previous data and plots (e.g., 99% percentile above), I’d be OK with deleting tweets that:

• Are older than 60 days
• Have less than 9 likes
• Have less than 4 retweets
• Are not self-liked by me

The command (and future cron job) will look like this:

# add --prune to switch from dry-run to deleting
plumes audit_tweets EngNadeau-tweets.json --min_likes 9 --min_likes 4 --days 60 --self_favorited False

This results in 1325 identified tweets that will be deleted. Goodbye :)

Analyzing and Pruning Friends

Next, let’s take a look at my friends (i.e., people I follow). Similar to the tweet analysis and pruning process, we’ll be using the plumes JSON output to get an idea of my following quality.

Loading Data

Like before, we will simply load the JSON data and convert to a DataFrame.

# load data
path = Path("EngNadeau-friends.json")
with open(path) as f:
    data = json.load(f)

# convert data to pandas dataframe
df = pd.json_normalize(data).pipe(
    lambda x: x.assign(**{"created_at": pd.to_datetime(x["created_at"])})
)

df

Like the tweets data before, the users JSON data is also nicely structured.

According to this (and my Twitter profile), I follow 1.4k+ people. I definitely don’t interact with that many people, nor pay attention to them. Let’s explore the data a little more and then clean this up.

Exploring Data

First, let’s take a look at basic user statistics:

df.filter(like="_count").describe()

Above we have a lot of useful info. From a segmentation perspective, followers_count followed by statuses_count gives the biggest variance if we wanted to classify our users into groups.

However, who are the super popular people that are outliers compared to the rest of my friends?

df.sort_values(by="followers_count", ascending=False).head()

Oh… hello Obama.

But this also brings up another issue: who should I actually follow?

As much as I like Obama and Bill Gates, I don’t actually interact with them. This is especially true for CNN and YouTube.

If there is something truly worthwhile being tweeted by these people (or orgs), I’ll probably hear about it from my thousand other social media sources. What I want from Twitter is more personal content from people that provide intelligent ideas and good discussion topics.

Let’s start by exploring a user’s followers vs. friends and their Twitter Follower-to-Friend (TFF ratio).

fig, ax = plt.subplots(figsize=figsize)

df.plot.scatter(x="friends_count", y="followers_count", ax=ax, label="Friend")

fig.suptitle("Friends' Followers vs. Friends")
ax.set_xlabel("Number of Friends")
ax.set_ylabel("Number of Followers")
ax.set_xlim([0, df["friends_count"].quantile(q=0.9)])
ax.set_ylim([0, df["followers_count"].quantile(q=0.9)])
ax.legend()
fig.tight_layout()

Right away, we can see that the majority of outliers fall below a TFF ratio of 1. Below TFF=1, people have many more followers than friends (e.g., celebrities, popular people). Above TFF=1, you follow a lot of people, but people don’t follow you back (e.g., up-and-comers, bots).

So who should I follow? Let’s look at who I’ve interacted with in the past.

First, I’ll load the my past retweets and favourited tweets (EngNadeau-favorites.json) data. Second, I’ll extract the users I’ve retweeted in the past. Third, I’ll compare these users vs. my current friends.

import numpy as np

# load retweet data
path = Path("EngNadeau-tweets.json")
with open(path) as f:
    data = json.load(f)

df_retweeted = (
    pd.json_normalize(data)
    .pipe(lambda x: x.assign(**{"created_at": pd.to_datetime(x["created_at"])}))
    .pipe(lambda x: x[x["retweeted"]])
    .filter(like="retweeted_status")
    .filter(like="_count")
    .pipe(
        lambda x: x.assign(
            ratio=x["retweeted_status.user.followers_count"]
            / x["retweeted_status.user.friends_count"]
        )
    )
    .replace([np.inf, -np.inf], np.nan)
    .dropna(subset=["ratio"])
)

df_retweeted.describe()

# load favourited tweets data
# run `plumes favorites`
path = Path("EngNadeau-favorites.json")
with open(path) as f:
    data = json.load(f)

df_favorites = (
    pd.json_normalize(data)
    .pipe(lambda x: x.assign(**{"created_at": pd.to_datetime(x["created_at"])}))
    .drop_duplicates(subset="user.screen_name")
    .filter(like="user.")
    .filter(like="_count")
    .pipe(lambda x: x.assign(ratio=x["user.followers_count"] / x["user.friends_count"]))
    .replace([np.inf, -np.inf], np.nan)
    .dropna(subset=["ratio"])
)

df_favorites.describe()

So, from the above, I typically interact with pretty popular people (retweeted_status.user.followers_count and user.followers_count), but most people fall within a TFF ratio of about 1 to 30.

What does this look like plotted?

fig, ax = plt.subplots(figsize=figsize)

df.plot.scatter(x="friends_count", y="followers_count", ax=ax, label="Friend")

df_retweeted.plot.scatter(
    x="retweeted_status.user.friends_count",
    y="retweeted_status.user.followers_count",
    ax=ax,
    label="Retweeted User",
    c="C1",
)

df_favorites.plot.scatter(
    x="user.friends_count", y="user.followers_count", ax=ax, label="Liked User", c="C2"
)

fig.suptitle("Users' Friends vs Followers")
ax.set_xlabel("Number of Friends")
ax.set_ylabel("Number of Followers")
ax.set_xlim([0, df["friends_count"].quantile(q=0.9)])
ax.set_ylim([0, df["followers_count"].quantile(q=0.7)])
ax.legend()
fig.tight_layout()

The vast majority of people I interact with are within the core grouping of less than 2k friends and 50k followers (i.e., a TFF ratio of 25).

Pruning Friends

With goal #2 in mind, we’ll use plumes to prune friends that I don’t typically interact with. From the previous data and plots, I’ll unfriend people that:

• Haven’t tweeted in the last 30 days
• Have a TFF ratio less than 1
• Have a TFF ratio more than 30

Since plumes assumes the conditional flags are AND boolean operations, we’ll need to call the bool_or flag to convert the boolean algebra to OR conditions. The command will look like this:

# add --prune to switch from dry-run to deleting
poetry run plumes audit_users EngNadeau-friends.json --min_ratio 1 --max_ratio 30 --days 30 --bool_or

This results in 912 identified users that will be unfriended.

My Twitter feed has never looked so good. :)