Programming Diary #23: Warning - this tool exercises your voting power

Conversation at the speed of Steem

Summary

This post covers the progress in the SCA during the last fortnight. In short, I think I'm on track for the goal of Open Sourcing it on September 2, and it's starting to be somewhat useful - though many defects remain. Through the use of this tool, I've become aware of much more organic activity on the blockchain and also of other interesting activity that was previously invisible to me.

The biggest drawback I have observed from using this tool is that it becomes increasingly difficult to manage my available voting power.

The post also reflects on the possibility for ecosystem participants to focus on tools and processes that create an attractive virtual force that pulls human attention into the ecosystem from other locations on the Internet. In my opinion, this can be accomplished through a combination of development efforts and better use of post promotion.

Background

It's hard to believe that two weeks have already passed since my last update, Programming Diary #22: Boosting organic conversations and reflecting on support for open source development. Time is a relentless companion.

As with other recent posts, my primary focus during the fortnight has been work on the Steem Conversation Accelerator (SCA).

Here were my short term goals from the previous post:

• During the next four weeks, I don't intend to add any new features. Instead, I'll focus on testing, debugging, code cleanup, and trying to make the activity page look cosmetically nicer.
• Once I'm satisfied with the cosmetics and the correct reporting of the currently intended functionality, I'll open source the repository.

And here are the git commits that have been posted during that time - 20 in all:

commit eaba7a6529aef8206a61bf70049338d2d1efe1f0

    - Changed calculation of fifteen minutes ago.

commit 70dc3bb515ad02df5c6d1c9e2100803d4a405797

    - Code review and cleanup.

commit 02ad938b6eaf3c28d11f8cdb6eacbba2a336d5e6

    - Changed to disable polling after 10 minutes of idle time, instead of immediately after going idle.

commit fa69a11740daad61fd0c973f6862e178670ca057

    More formmatting.

commit efd54d2450993b77d515dffd4d01f0cfcfa4d38b

    - Playing around with coloring and formatting.
    - Added some network error checking.
    - Code simplification/refactoring with claude.ai.

commit 4d3b7e84375b3b6302e7ccc62e47034d12e5a6f9

    Code simplification and changed account_history calls to grab 20 at a time.

commit c53411434a4c77d1d13366be2cea9521df5dba35

    Simplifications for readability.

commit c7a4b62823dc32f744af7bf0a67c40932a053197

    1.) Time stamps and post displays are behaving sort-of rationally, now.

    I'm pretty sure there are still race conditions to be checked.  Especially around lock preemption when the activityList script preempts the background script.

    2.) Fixed a bug in deleteTriplet() that was causing deletions to fail.

commit efb9fb3e00d9e3f75d9fa72934225c1b129d63a1

    Monkeying around with timestamp logic again in order to get rid of misses and repeats.

commit 6f7306848aab703631ebdc9466aa31c672474a3b

    Troubleshooting issues with missing/duplicated entries for posts, comments, and replies.

commit 7e02d5df37dcf8cc642e2b5e6fdbba09abccc6d7


    More HTML and CSS updates.

commit 30d46613089fd89cbc55aaa289630d0c06eea678

    Added .gitignore to exlude .vs (visual studio) files.

commit 502ec70fe8b0ab6260b5d3f1f2550854f610794f

    Playing with coloring and added collapsible sections.

commit a7c1876a40bfa7bbcb34628856d43094433d1c7a

    Formatting changes.  Commenting out (apparently) unused code.

commit d5bdf55c913d6e3bfcb18722b62ca1c745ec71ac

    - HTML page layout changes
    - Changed errors to warnings for retryable operations.

commit f45a0b32b07ce3f75e772a2088711dd8a937b12c

    Continued refactoring & debugging.  Saving at breakpoint.

commit fe351b353bc6f936a9bccb5a4ef1140dad0aa6ba
Merge: 55075bd 1550cbb

    Merge branch 'formatHTML' of https://github.com/remlaps/Steem-Conversation-Accelerator-SCA- into formatHTML

commit 55075bdff2708fe5cd118a54fdec10742624932b

    First round of refactoring complete.  The same code is now in use by
       posts, comments, and replies.

commit 1550cbb28bd5cac8bb0b017eca6ecbafe504a3e8

    First round of refactoring complete.  The same code is now in use by
       posts, comments, and replies.

commit e13c3d31ecc11d1d6fb6a91a0bf16d72c8b3fe31

    Moved update logic for readability.  Preparing for refactoring.  Saving interrim state.

(it's sort of sad to see the careless typos that are saved forever in "git" comments, and now highlighted by a spell checker. 😞)

Basically, I stayed on track with my goals, but I also got more involved than I had expected in network changes and speed improvements. Including time spent debugging a network problem that wasn't really a problem, and switching from Netbeans to vsCode for my IDE. Primary themes for the fortnight:

• Code reorganization
• Network performance, including API endpoints
• Display format changes

So, let's review them.

Activity Descriptions

Code reorganization - including timestamps and idle time

When I was working on my first master's degree, I had a professor who said that when you have a programming assignment, you should write the code once to learn how to do it, and then throw out your work and redo it "the right way". I'm not going to go through that exercise, but there was a lot of truth in his logic. I have reorganized things several times while learning about the unique interactions of the browser, the browser extensions, and the API endpoints.

One of my biggest choke points has been on the handling of the logic for tracking posting time and polling time. What I settled in on was something like this:

1. For each followed account, I record the following information:
   1.1. followed account name
   1.2. The last time that the account's posts/comments/replies were displayed in the activity list
   1.3. The last time that the author posted/received a post, comment, or reply.

Then, whenever the last activity time is newer than the last displayed time, I display the account's new activity. Simple, right?

Unfortunately, not so much. This simple flow is disrupted by the fact that there are three components that need to be kept in synch with user behavior.

• When all accounts have been checked by the background script, if there is new activity, then a browser notification is sent. The browser operator might click on that notification immediately, or they might delay, which would give the background script time to start polling again.
• When the browser operator clicks on the browser notification and opens the activity list HTML page, that may or may not remain open while the next pass through the background script happens.
• The account_history data coming back from the API endpoints can be 10 minutes or more behind the actual clock time.

All of this makes the logic complicated, and I'm still fairly confident that there are mistakes to be corrected, but it's getting closer. And, it's usable, even if not entirely correct.

Network performance, including API endpoints

Network performance

At the beginning of the fortnight, out of six that I had tried, there were only two API endpoints that I could use for this browser extension: api.moecki.online, from @moecki; and api.steememory.com, from @yasu. The Steemit endpoints (api.steemit.com, and api.steemitdev.com) were unusable due to rate limiting, and the other two that I tried (api.pennsif.net / @pennsif, and steemd.steemworld.org / @steemchiller) are not running the followers plugin, so I don't know a way to generate the list of followed accounts that I need. (side note I'm also not sure why condenser_api/get_following seems to need the followers plugin. That seems odd.)

IMO, two endpoint choices is not enough for an app like this, so I was frustrated. Because of the development history, my previous iteration was batching the transaction history one at a time, and it turns out that this is why Steemit's (really slow) rate limiting was being triggered. By grabbing 20 transactions at a time and then iterating through the list, I was able to avoid most of the throttling, so the extension now has four usable API endpoints.

Debugging a phantom network problem

While working on the account_history changes, I was also having problems with intermittent network failures, and I was hopelessly stumped. This was also alluded to in the previous Programming Diary post:

I started working on some other tasks like cleaning up code that was obsolete or inserted for debugging purposes, and when I did, network functions that were previously working started failing again...In certain situations, it seems that my code was "hallucinating" network errors, and bailing out when it did.

I chatted about it with Google Gemini, ChatGPT, and Claude, but they were all stumped too.

I didn't get a break until I happened to notice that I was triggering the problem myself when I switched to the Network, Performance, Memory, or Application tabs in the DevTools environment that I was using for troubleshooting. Like this:

Even that observation didn't let me solve the problem immediately, but eventually I noticed that the same thing didn't happen in MS Edge, or even in other Profile windows of the same Brave browser. From that, I finally looked closely enough to see that there's an option in the DevTools window to simulate throttling, and I had inadvertently flipped that to "Offline". So, whenever I changed to one of those tabs, the browser started pretending that it was offline.

Didn't I feel foolish. 😆

An interesting phenomenon of AI is that - after spending so much time chatting with the AIs about it - I had a strange feeling like I should "close the loop" and tell Claude what the problem was when I finally figured it out.

Display format changes

Not much to talk about here, just that I played around with coloring and layout, and now the SCA activity list looks like this:

Note the new tag-line, "Conversation at the speed of Steem". Should I keep it?

I'm pretty sure this format will change again before I release it as open source, but at least now it's not embarrassingly bad any more.

You can see that the sections are now collapsible, and I also added clickable links to the account's page, the author's page, and the relevant "Posts/Comments/Replies" sections. Finally, I added an icon to act as a logo in the upper-left corner.

Timestamps and idle time

The problem here is dealing with the interactions between all of the components. It would be simple if there were just a list of transactions coming in and a list of activity going out, but we also have to consider the variable lag in blockchain reporting, the dynamic behavior of the browser operator, and the handoff of information between components of the browser extension.

So, the easiest way to deal with the complexity is to slow down the polling time. That way, there's less interaction between the activity list page and the background polling.

This raises two new problems, though. (i) Less frequent polling means later notifications; and (ii) The extension was configured to go to sleep if the browser goes idle, and the browser goes idle very quickly.

The first problem was balanced out by the network improvements from the first section, so that's not really a concern right now. Even when it polls at ten minutes instead of one minute, the overall collection time is so much faster that the full cycle seems to be faster.

For the second problem, though, if the browser operator switches between apps or steps away from their keyboard, it was possible that the browser might never be active for long enough to complete a polling cycle. To mitigate this, I set a ten minute timer. Now, if the browser flips to idle and stays that way for ten minutes or more, then the extension goes to sleep. If it is only idle for a short time, however, polling doesn't get interrupted.

Next up

This is basically unchanged from last time. I'm still hoping to open source the repo by Labor Day - September 2 - with a minimally (VERY minimal) functioning browser extension, but I already know that my time is limited for this coming week, so we'll see.

• During the next four two weeks, I don't intend to add any new features. Instead, I'll focus on testing, debugging, code cleanup, and trying to make the activity page look cosmetically nicer.
• Once I'm satisfied with the cosmetics and the correct reporting of the currently intended functionality, I'll open source the repository.
• After that, I'll add the following additional features:
  • Notifications for replies on a conversation that the observer has participated in.
  • Notifications for selected tags/communities
  • Filtering of accounts and/or notification types
  • Displaying the Follower Network Strength and reputation scores of authors who appear in the activity list
  • Incorporating Visibility As A Service into the displayed activity list.

Somehow, I also have to find time to look at Q3 adjustments to the Follower Network Strength calculation for my automated voter and for the Steem Follower Checker.

Reflections

What do I have for reflections this time? Not much, really. The suggestion to launch a Steem's Best Open Source Incubator didn't receive as much feedback as I might have hoped, and there haven't been any volunteers to help develop it, so... I still think it's a good idea, but right now my thoughts have been buried in the SCA, and time is limited. Maybe some more thoughts about next steps on that in 2 or 3 weeks. Overall, I'm imagining a timeline where something launches in 2025 and builds capabilities over several years.

I'm definitely happy to see some progress at reducing the missed blocks by Steem witnesses.

After pausing for a moment, I do have a couple topics that we can tie together into a general theme of "attraction".

First, let's recall the post from @o1eh: Unexploited Opportunity: Post Promotion, where @o1eh pointed out that the blockchain has the capability to offer post promotion, but it's hidden away so that few know about, and it brings little benefit for the people who do use it. He rightly notes that Steem interfaces (like condenser) could boost the ecosystem by giving more visibility to /promoted posts. But, what if that's not actually the best use of post promotion? What if the best use is something that we haven't thought much about?

Let's revisit a theme from several previous programming diaries:

When people are active in their browsers on other web sites, notifications from the SCA may bring their attention back to the Steem blockchain. Previously, I described this as a virtual analog to gravity in the physical world. In particular, I have noticed that when blog authors embed X (Twitter) posts in their articles, I frequently discover myself back on the Twitter web site, even when I had no intention of visiting there.

X (Twitter) is powerful because those little embeds create a huge surface area that draws people back to the platform from almost all corners of the Internet. This is one idea behind the SCA, drawing people from their other browser activities back to the Steem blockchain. Maybe post promotion could serve a similar purpose?

I already touched on this in Programming Diary #9: Steem + Informatics = Steemformatics,

The information provided by the app doesn't even have to be about Steem. It could be something as simple as stock prices or weather reports. As long as it also provides visibility for token-burning, it would still benefit the blockchain's users. For example, imagine a widget on CoinMarketCap, Yahoo Finance, or TronScan with a scrolling list of Steem's #burnsteem25 posts.

but you never know when a spark will start a brushfire. Imagine a link between post promotion on Steem and visibility on some external web site. Then, not only would it provide visibility to the post's author, but it would also bring visitors to the web site.

Could this be done in partnership with the @tronfoundation or Brave or @coingecko or @officialcmc? Could it be funded by the SPS? For example, "Promote your post with SBDs above some minimum, and double that amount will go from the SPS to a partner's platform in exchange for off-chain visibility of the post (and blockchain - subject to content/quality review)."?

Conclusion

In all, progress has been decent on the SCA this week, and I think I'll probably be able to Open Source it (with defects) in two weeks, as planned. Although the browser extension is far from perfect, I have found that it's already useful.

Just by running it during this development phase, I have become newly aware of many conversations and other aspects of the blockchain that were previously hidden from me, and I have been able to provide more votes for organic activity with less overall effort. The main challenge that this brings is that it becomes difficult to manage my voting power, and I frequently find myself below 50%.

In addition to benefitting the browser operator, I think it also benefits the Steem ecosystem by bringing human attention back to the platform more frequently.

Lastly, this post discussed two longer term initiatives that I think can be useful to the ecosystem - Incubating the Open Source Development community, and improving the link between post promotion and visibility for both the author and the blockchain.

---

Thank you for your time and attention.

As a general rule, I up-vote comments that demonstrate "proof of reading".

---

Steve Palmer is an IT professional with three decades of professional experience in data communications and information systems. He holds a bachelor's degree in mathematics, a master's degree in computer science, and a master's degree in information systems and technology management. He has been awarded 3 US patents.

---

^{Pixabay license, source}

Reminder

Visit the /promoted page and #burnsteem25 to support the inflation-fighters who are helping to enable decentralized regulation of Steem token supply growth.