使用 partition by 查找并删除 MySQL 数据库中重复的行

Posted on 2020年1月20日2020年1月21日 by 逸飞

在创建 MySQL 数据表的时候，经常会忘记给某个字段添加 unique 索引，但是等到想添加的时候又已经有了重复数据，这时候就需要删除重复数据。

准备数据

本文使用如下的数据作为演示：

CREATE TABLE contacts (
    id INT PRIMARY KEY AUTO_INCREMENT,
    first_name VARCHAR(50) NOT NULL,
    last_name VARCHAR(50) NOT NULL,
    email VARCHAR(255) NOT NULL
);

INSERT INTO contacts (first_name,last_name,email) 
VALUES ('Carine ','Schmitt','carine.schmitt@verizon.net'),
       ('Jean','King','jean.king@me.com'),
       ('Peter','Ferguson','peter.ferguson@google.com'),
       ('Janine ','Labrune','janine.labrune@aol.com'),
       ('Jonas ','Bergulfsen','jonas.bergulfsen@mac.com'),
       ('Janine ','Labrune','janine.labrune@aol.com'),
       ('Susan','Nelson','susan.nelson@comcast.net'),
       ('Zbyszek ','Piestrzeniewicz','zbyszek.piestrzeniewicz@att.net'),
       ('Roland','Keitel','roland.keitel@yahoo.com'),
       ('Julie','Murphy','julie.murphy@yahoo.com'),
       ('Kwai','Lee','kwai.lee@google.com'),
       ('Jean','King','jean.king@me.com'),
       ('Susan','Nelson','susan.nelson@comcast.net'),
       ('Roland','Keitel','roland.keitel@yahoo.com'),
       ('Roland','Keitel','roland.keitel@yahoo.com');

注意其中有一行重复了三次。输入完成后，数据如图所示：

file

查找重复的行

使用 group by 和 having

假设我们要通过 email 字段来查找重复值。通过使用 group by 和 having 子句可以查找到哪些行是重复的。

SELECT
    email,
    COUNT(email)
FROM
    contacts
GROUP BY email
HAVING COUNT(email) > 1;

file

Having 就类似于 Group by 之后的 where 子句。但是这个语句还是很难解决我们的问题，我们只知道发生重复的第一行了，而不知道哪些行是重复的。这时候可以使用 partition by 语句。

使用 partition by 找出所有的重复行

需要注意的是，partition by 只有在 MySQL 8.0 之后才支持，而现在常用的是 5.6 和 5.7 版本。

Partition 语句又叫做窗口聚合语句，也就是说他会把同一个值的行聚合成一个窗口，但是和 Group by 语句不同的是，窗口内的每一个行并没有被压缩成一行，具体说Partition by 的语法是：

window_function_name(expression) 
    OVER (
        [partition_defintion]
        [order_definition]
        [frame_definition]
    )

删除重复的行

删除的方法有很多种，这里介绍两种。

References

https://www.mysqltutorial.org/mysql-window-functions/

But the rumor, sell the fact.

Posted on 2020年1月17日 by 逸飞

buy the rumor, sell the fact 说的是市场往往具有领先性，在事情还没有落地之前，已经开始抢跑了。等到协议真正签署的时候，也就是利好兑现的时候，当前的市值早已经蕴含了对这个利好成功兑现的预期。所以在真正达成协议的时候，先期在低价买入的一方便会决定卖出以兑现自己的收益。由于这时候卖方变多，如果没有后续利好的情况下，利好兑现反而会导致股价会出现一定的回调。反过来说，有时候利空出尽甚至会带来股价的上扬，当然也是在基本面看好的前提下。举几个例子就很明白啦~

超出市场预期的例子。中美谈判对峙时期，美国态度强硬。Xi 突然去江西视察了一家稀土企业——金力永磁，并支持稀土是中国的战略资产，必要时候可以限制出口。很明显这是针对美国的嘛，因为这个事情之前市场都没有想到过，原来稀土还能有这么大的价值，所以这就是一件超出市场预期的事情，那么包括金力永磁、宁波韵升之类的一大批稀土企业的股票就都暴涨了。http://finance.sina.com.cn/stock … vhiqay3582814.shtml
不及市场预期的例子。拼多多三季报营收大增 122%，亏损也扩大了 112.6%，总的来说这份财报有喜有忧，但是股票直接从 $40 掉到了 $30。主要原因是不及预期，也就是说市场之前没想着亏损竟然能这么大，当时的市值已经高估了拼多多的盈利能力。https://xueqiu.com/7700511931/136081485
超出市场预期的例子。格力的股权交易。由于格力的管理层董明珠和大股东珠海国资委的关系一向不是很好，所以格力的控股权交易让市场充满了担忧。按照 buy the rumor 的说法，在格力被收购的传闻阶段股价就会被抬起来了，但是因为刚刚说的原因，股价反而一直维持在低位。直到靴子落地，高瓴资本正式和管理层签订了协议，没有发生撕逼的事情，格力才开始从 50 多块一路涨到了现在的接近 70。
利好兑现的例子。这个就太多了，刚开始炒股的时候一看这个年报也不错，那个季报也挺牛逼，觉得买买买就好了，往往就容易入套了。因为一个公司的经营状况不可能是完全保密的，比如说线上的销售数据啊、苹果的出货量啊、原材料的价格啊这些数据只要用心或者花钱都是可以收集到的，那么其实机构们早在报表发布之前已经算准了你盈利到底增加多少。如果存在低估现象的话，早就提前入场了，如果报表符合预期（大概率），那么可能就会确定收益。最近我持仓的一个例子：国联股份，1月9号公布了业绩预增61%-70%，但是随后两天却从最高点还是阴跌了，这就是利好兑现的一个典型。http://stock.jrj.com.cn/2020/01/09192728651513.shtml

回到中美协议这个事情，一方面谈了这么久，早就确定要签了，协议的内容其实两边也都基本 ok，这也就是大家都知道的事情。另一方面，Trump 面临 2020 的大选，是在是不太可能有精力在大选前在折腾了。所以这件事情虽然是个很大的利好，但是不管 A 股还是美股，早就先涨为敬了。协议的落地长远来看肯定是很大的利好，对短期来说，说不定是利空呢。。

所以最重要的有时候不是利好还是利空，重要的是有没有超出或者不及市场（分析师）的预期。市场的预期在哪里找呢？可以先看看券商的研报，一般都有某个股票的目标价位。

Puppeteer 中如何绕过无头浏览器检测

Posted on 2019年12月20日 by 逸飞

执行以下代码：

# credits: https://intoli.com/blog/making-chrome-headless-undetectable/

import pyppeteer as pp

from futile2.http import get_random_desktop_ua

HIDE_SCRIPTS = dict(
    hide_webdriver="""
() => {
    Object.defineProperty(navigator, 'webdriver', {
      get: () => false,
    });
  }
""",
    hide_navigator="""
() => {
    // We can mock this in as much depth as we need for the test.
    window.navigator.chrome = {
      app: {
        isInstalled: false,
      },
      webstore: {
        onInstallStageChanged: {},
        onDownloadProgress: {},
      },
      runtime: {
        PlatformOs: {
          MAC: 'mac',
          WIN: 'win',
          ANDROID: 'android',
          CROS: 'cros',
          LINUX: 'linux',
          OPENBSD: 'openbsd',
        },
        PlatformArch: {
          ARM: 'arm',
          X86_32: 'x86-32',
          X86_64: 'x86-64',
        },
        PlatformNaclArch: {
          ARM: 'arm',
          X86_32: 'x86-32',
          X86_64: 'x86-64',
        },
        RequestUpdateCheckStatus: {
          THROTTLED: 'throttled',
          NO_UPDATE: 'no_update',
          UPDATE_AVAILABLE: 'update_available',
        },
        OnInstalledReason: {
          INSTALL: 'install',
          UPDATE: 'update',
          CHROME_UPDATE: 'chrome_update',
          SHARED_MODULE_UPDATE: 'shared_module_update',
        },
        OnRestartRequiredReason: {
          APP_UPDATE: 'app_update',
          OS_UPDATE: 'os_update',
          PERIODIC: 'periodic',
        },
      },
    };
  }
""",
    hide_permission="""
() => {
    const originalQuery = window.navigator.permissions.query;
    return window.navigator.permissions.query = (parameters) => (
      parameters.name === 'notifications' ?
        Promise.resolve({ state: Notification.permission }) :
        originalQuery(parameters)
    );
  }
""",
    hide_plugins_length="""
() => {
    // Overwrite the `plugins` property to use a custom getter.
    Object.defineProperty(navigator, 'plugins', {
      // This just needs to have `length > 0` for the current test,
      // but we could mock the plugins too if necessary.
      get: () => [1, 2, 3, 4, 5],
    });
  }
""",
    hide_language="""
() => {
    // Overwrite the `plugins` property to use a custom getter.
    Object.defineProperty(navigator, 'languages', {
      get: () => ['en-US', 'en'],
    });
  }
""",
    hide_webgl_renderer="""
() => {
    const getParameter = WebGLRenderingContext.getParameter;
    WebGLRenderingContext.prototype.getParameter = function(parameter) {
      // UNMASKED_VENDOR_WEBGL
      if (parameter === 37445) {
        return 'Intel Open Source Technology Center';
      }
      // UNMASKED_RENDERER_WEBGL
      if (parameter === 37446) {
        return 'Mesa DRI Intel(R) Ivybridge Mobile ';
      }

      return getParameter(parameter);
    };
}
""",
    hide_broken_image="""
() => {
    ['height', 'width'].forEach(property => {
      // store the existing descriptor
      const imageDescriptor = Object.getOwnPropertyDescriptor(HTMLImageElement.prototype, property);

      // redefine the property with a patched descriptor
      Object.defineProperty(HTMLImageElement.prototype, property, {
        ...imageDescriptor,
        get: function() {
          // return an arbitrary non-zero dimension if the image failed to load
          if (this.complete && this.naturalHeight == 0) {
            return 20;
          }
          // otherwise, return the actual dimension
          return imageDescriptor.get.apply(this);
        },
      });
  });
}
""",
    hide_modernizr="""
() => {
    // store the existing descriptor
    const elementDescriptor = Object.getOwnPropertyDescriptor(HTMLElement.prototype, 'offsetHeight');

    // redefine the property with a patched descriptor
    Object.defineProperty(HTMLDivElement.prototype, 'offsetHeight', {
      ...elementDescriptor,
      get: function() {
        if (this.id === 'modernizr') {
            return 1;
        }
        return elementDescriptor.get.apply(this);
      },
    });
}
""",
)

async def get_headless_page(*args, **kwargs):
    """
    生成一个无法检测的浏览器页面
    """
    browser = await pp.launch(*args, **kwargs)
    page = await browser.newPage()
    await page.setUserAgent(get_random_desktop_ua())
    for script in HIDE_SCRIPTS.values():
        await page.evaluateOnNewDocument(script)

    return page

为什么不使用 scrapy，而是从头编写爬虫系统？

Posted on 2019年12月12日2019年12月12日 by 逸飞

时隔一年了，来回答下自己提的问题。个人不喜欢 scrapy 原因一言以蔽之：高不成，低不就，弊大于利。
总的来说，需要使用代码来爬一些数据的大概分为两类人：

非程序员，需要爬一些数据来做毕业设计、市场调研等等，他们可能连 Python 都不是很熟；
程序员，需要设计大规模、分布式、高稳定性的爬虫系统，对他们来说，语言都无所谓的，更别说用不用框架了。

为什么不适合初学者？

对于初学者来说用不上 scrapy 的原因很简单：

scrapy 太复杂了；
scrapy 采用异步模式带来的高性能和在反爬面前实际上没有任何卵用；
scrapy 项目冗余的代码结构对初学者完全是过度设计。

对于一个任何一个已经入门的程序员来说，Python 都算不上一个很复杂的语言，除了不用大括号可能让一些人感觉有些不适应之外，基本上看看语法上手就能写了。但是恰恰是因为我们都是老司机了，所以不能体会到使用一门编程语言对于外行来说可能『比登天还难』。如果不用 scrapy，可能我只需要这样：

# 以下代码未经测试，可能有些许 bug
import requests

def main():
    for i in range(100):
        rsp = requests.get(f"http://www.example.com/{i}.html")
        with open("example-{i}.html", "w") as f:
            print(f"saving {i}")
            f.write(rsp.text)

if __name__ == "__main__":
    main()

就写好了一个简单的爬虫。而使用 scrapy 呢，大概需要这样吧：

# 以下代码未经测试，可能有些许 bug
import scrapy

class QuotesSpider(scrapy.Spider):
    name = 'quotes'

    def start_requests(self):
        for i in range(100):
            yield scrapy.Request(url=f"http://www.example.com/{i}.html", callback=self.parse)

    def parse(self, response):
        page = response.url.split('/')[-2]
        with open('example-%s.html' % page, 'wb') as f:
            f.write(response.body)
        self.log('Save file %s' % page)

先不说代码增长了不少，初学者会问到这些问题：“什么是 class？为什么类还有参数？啊，什么是继承？yield 又是什么鬼，那个 scrapy.Request 又是啥？”这些都是心智负担。那么 scrapy 这些心智负担又给我们带来了什么好处呢？好处是性能和相对来说比较统一的代码结构，但是其实这两个对初学者并没有什么卵用啊……

scrapy 采用了 twisted 作为基础，实现了基于协程的高并发。协程看着虽然挺好，但是对于非程序员来说，他们往往就想对一个站点做定向爬取，你说你蹭蹭蹭把并发涨上去了，无非两个后果：

对方承受不住你爬，挂掉了，你拿不到数据；
对方把你封禁了，疯狂弹验证码，你拿不到数据。

所以，对于非程序员做的一些定向爬取来说，速度是没有意义的，甚至往往是越慢越好。scrapy out。

那么相对来说比较统一的代码结构有什么卵用吗？答案依然是没有。我们知道在 web 开发领域基本上稍微有点规模的项目还是要使用框架的，哪怕是 flask 这种微框架。在 web 开发领域，有经典的 MVC 模式，我们需要路由、模板、ORM 这些固定的组件，所以主循环是由框架和 web server 来控制的。而对于爬虫呢？其实没有什么固定的模式，scrapy 也仅仅是定义了几个钩子函数而已，反倒我们没有了主循环，在编写一些特定逻辑的时候非常受到掣肘。

另外 scrapy 提供的一些其他功能，比如说抓取的队列或者去重等等，个人感觉有过度封装的味道，而且也都是在内存里，在反爬导致爬虫挂掉这种故障面前没有什么卵用，不二次开发的话还是得重爬。对于小白来说，也不用想 redis 这些幺蛾子，其实可以用 Google 最开始使用的一个很简单的方法，就把每个新抓到的 url 写到一个 txt 文件就好了，爬虫每次重启的时候首先读取这个 txt 就好了，网上乱七八糟的教程大多是炫技的。

为什么不适合大型爬虫系统？

前面说到，scrapy 基于 twisted。twisted 是 Python 的一个异步框架，最大的问题就是太难懂了，而且现在官方应支持了 asyncio，所以 twisted 的未来堪忧，甚至比起 twisted 来说，我更愿意投入时间到 curio 这样新兴的有潜力的异步框架。第二点就是 scrapy 控制了主循环，所以二次开发相当于只能在他的框架内做一些修修补补，并且还要兼容 twisted。

既然要开发大型爬虫系统，那么其中很重要的一部分就是爬虫的调度了。一种比较简单的模式是 scheduler 作为 master，全局调度。另一种模式没有 master，所有的爬虫 worker 都是对等的。在实际生产中显然是第一种用的更多。

显然 scheduler 这部分是不能再用一个爬虫框架来实现的，连主循环都没有怎么写逻辑呢？我们可能还要实现增量爬取，或者消费业务方发来的爬取请求等各种业务，这块显然是在 scheduler 里面的，那么这个爬虫系统无非是 scheduler 分发任务给各个 worker 来抓取。worker 还可以使用 scrapy 实现，但是呢，这个 worker 其实已经弱化为一层薄薄的 downloader 了，那我要他干嘛呢？scrapy 的核心逻辑也不过是个深度或者广度优先的遍历而已，少一个依赖不好么……

总结一下，爬虫的工作量要么在反爬，要么在调度等业务逻辑，本身只是一个 requests.get 而已，scrapy 提供的种种抽象对于初学者太复杂，大型系统又用不上，所以个人不推荐使用包括但不限于 scrapy 在内的所有爬虫框架。

建议所有认为学习框架会使自己变强的人读读：Stop learning frameworks 和评论，中文翻译

以上仅代表个人观点，欢迎讨论，不要人身攻击。

如何在 URL 中表示数组

Posted on 2019年12月5日2019年12月7日 by 逸飞

我们知道 URL 后面的 query string 实际上是一个字典的形式。URL 的任何一个规范中都没有定义如何在 query 中传递数组，但是这个需求也是实际存在的，于是就诞生各种奇葩的形式，本文做一个总结。

常见的形式

http://www.baidu.com/search?q=url&tag=foo

这是一个正常的 URL，这里解析出来应该是一个字典 {"q": "url", "foo": "bar"}。但是 Python 会强行解析成数组 {"q": ["url"], "tag": ["foo"]}。

使用 URL 表示数组有以下几种常见形式：

http://www.baidu.com/search?q=url&tag=foo&tag=bar

重复键表示数组，Python/Node 中可以正确解析成数组，Java 只读取第一个值，PHP 只读取最后一个值。

http://www.baidu.com/search?q=url&tag[]=foo&tag[]=bar

键后增加[]并重复表示数组。PHP/Node 可以解析为 tag=[foo, bar]。Python 会解析成

PHP 的 http_build_query 会生成这种格式。

In [6]: from urllib.parse import parse_qs

In [7]: parse_qs("tag=foo&tag=bar")
Out[7]: {'tag': ['foo', 'bar']}

In [8]: parse_qs("tag[]=foo&tag[]=bar")
Out[8]: {'tag[]': ['foo', 'bar']}

In [9]: parse_qs("tag=foo")
Out[9]: {'tag': ['foo']}

http://www.baidu.com/search?q=url&tag[0]=foo&tag[1]=bar

使用数组形式表示。貌似没有原因能够处理，但是用的还挺多的。

http://www.baidu.com/search?q=url&tag=foo,bar

使用逗号分隔。貌似没有语言默认会处理这种，需要自己手工处理。但是我最喜欢这种。

一个更奇葩的例子

https://www.doi.gov/careers/explore-careers?f[0]=bureaus:20&f[1]=competencies:1638&f[2]=competencies:1642&f[3]=competencies:1648&f[4]=competencies:1656&f[5]=competencies:1661&f[6]=gs_levels:17&f[7]=gs_levels:158

总之，在不同的语言中，乃至于不同的 web 框架中对以上形式有不同的解析，非常混乱。

参考资料

Python metaclass 的原理和应用

Posted on 2019年12月2日2019年12月2日 by 逸飞

元编程(meta programming)是一项很神奇的能力，可以通过代码在运行时动态生成代码。元类(meta classes)是 Python 提供的一种元编程的能力。在 Python 中，类也是一种对象，那么类这种对象就是元类的实例，所以我们可以在运行时通过实例化元类动态生成类。

使用 type “函数”

首先我们来了解一下 type，type 可以作为函数使用，用来获得对象的类型：

>>> class Foo:
...     pass
>>> obj = Foo()
>>> obj.__class__
<class '__main__.Foo'>
>>> type(obj)
<class '__main__.Foo'>
>>> obj.__class__ is type(obj)
True

实际上 type 并不是一个函数，而是一个类，我们可以使用 type(type) 来确定一下：

>>> type(type)
<class 'type'>

type 实际上不只是类，而是一个“元类”。我们接下来要可以看到，所有的元类都需要继承自 type。type 是所以类的元类，所以在上面的例子中 x 是 Foo 的实例，Foo 是 type 的实例，type 又是他自己的实例。

file

使用 type 动态创建类

如果传递给 type 的参数是三个的时候，type 的语义就不再是返回给定参数的类，而是实例化生成一个新的类。

type(name: str, bases: tuple, namespace: dict)

第一个参数是新生成的类的名字；第二个参数是新生成的类的基类列表；第三个参数是要个这个类绑定的属性的列表，比如说这个类的一些方法。实际上 class Foo 这种语法只是使用 type 生成类的语法糖而已。

最简单的一个例子，比如我们要创建 Foo[0..9] 这些类，可以这样做：

classes = []
for i in range(10):
    cls = type("Foo%s" % i, tuple(), {})
    classes.append(cls)

# 就像使用普通类一样初始化 Foo0

foo0  = clssses[0]()

如果要实现类的方法，一定要记得同样是要使用 self 变量的。在 Python 中 self 只是一个约定俗称的变量，而不是关键字。

def __init__(self, name):
    self.name = name

def print_name(self):
    print(self.name)

Duck = type("Duck", tuple(), {"__init__": __init__, "print_name": print_name})

duck = Duck("Donald")

duck.print_name()
# Donald

创建自己的元类

首先我们来回顾一下 Python 中类的初始化过程：

foo = Foo()

当这条语句运行的时候，Python 会依次调用 __new__ 和 __init__ 方法。其中 __new__ 方法在 __init__ 之前调用，并返回已经创建好的新对象，而 __init__ 函数是没有返回结果的。一般情况下，我们都会覆盖 __init__ 方法来对新创建的对象做一些初始化操作。

现在回归到元类上，进入烧脑部分。前面我们说过元类的实例化就是类，所以大致相当于：

Foo = MetaFoo(name, bases, attrs)  # MetaFoo 默认情况下是 type
foo = Foo()

默认情况下，所有类的元类是 type，也就是在这个类是通过 type 来创建的，这和前面说的通过 type 来动态创建类也是一致的。

那么怎样定义一个 MetaFoo 呢？只需要继承自 type 就行了。因为元类的实例化就是类的创建过程，所以在元类中，我们可以修改 __new__ 来在 __init__ 之前对新创建的类做一些操作。

>>> class MetaFoo(type):
...     def __new__(cls, name, bases, namespace):
...         x = super().__new__(cls, name, bases, namespace)  # super实际上就是 type
...         x.bar = 100  # 为这个类增加一个属性
...         return x
...

>>> Foo = MetaFoo("Foo", tuple(), {})  # MetaFoo 在这里就相当于 type 了，可以动态创建类
>>> Foo.bar
100
>>> foo = Foo()
>>> foo.bar
100

在这里我们创建了 MetaFoo 这个元类，他会给新创建的类增加一个叫做 bar 的属性。

在实际的代码中，我们一般还是不会直接动态生成类的，还是调用 class Foo 语法来生成类比较常见一点，这时候可以指定 metaclass 参数就好了。可以通过 Foo(metaclass=MetaFoo) 这种方式来指定元类。

class Foo(metaclass=MetaFoo):
    pass

这种定义和上面的元类用法效果完全是一致的。

一个现实世界的元类例子

在 django.models 或者 peewee 等 ORM 中，我们一般使用类的成员变量来定义字段，这里就用到了元类。

class Field:
    pass

class IntegerField(Field):
    pass

class CharField(Field):
    pass

class MetaModel(type):
    def __new__(meta, name, bases, attrs):
        # 这里最神奇的是：用户定义的类中的 bases 和 attrs 都会作为参数传递进来
        fields = {}
        for key, value in attrs.items():
            if isinstance(value, Field):
                value.name = '%s.%s' % (name, key)
                fields[key] = value
        for base in bases:
            if hasattr(base, '_fields'):
                fields.update(base._fields)
        attrs['_fields'] = fields
        return type.__new__(meta, name, bases, attrs)

class Model(metaclass=MetaModel):
    pass

这样用户使用的时候就可以这样定义：

>>> class A(Model):
...     foo = IntegerField()
...
>>> class B(A):
...     bar = CharField()
...
>>> B._fields
{'foo': Integer('A.foo'), 'bar': String('B.bar')}

程序在执行的时候就可以直接访问 X._fields，而不用每次都通过反射遍历一次，从而提高效率以及做一些验证。

不过，其实这个完全可以通过装饰器来实现：

def model(cls):
    fields = {}
    for key, value in vars(cls).items():
        if isinstance(value, Field):
            value.name = '%s.%s' % (cls.__name__, key)
            fields[key] = value
    for base in cls.__bases__:
        if hasattr(base, '_fields'):
            fields.update(base._fields)
    cls._fields = fields
    return cls

@model
class A():
    foo = IntegerField()

class B(A):
    bar = CharField()

但是用装饰器的话，就失去了一些类型继承的语义信息。

总结与思考

Python 中的元编程还是一种很强大的特性，但是也比较复杂，有时候很难以理解。实际上，过分的动态特性也导致了 Python 的解释器和静态分析、自动补全等很难优化，因为有好多信息必须到运行时才能知道。

实际上近些年新开发的语言越来越多地加入了静态类型的特性，比如 swift, rust, go 等。就连 Python 本身也增加了 type hinting 的功能，很遗憾的是，这个功能不是强制性的，所以也很难用来提升性能。

元类这块应该是我在 Python 语言方面了解的最后一大块知识了。接下来除了写业务代码不会再深究 Python 了，研究 Golang 去了~

Au revoir, Python!

参考

给 Python 程序员的 Go 语言教程

Posted on 2019年11月26日2019年11月29日 by 逸飞

楔子

最近读到一亩三分地上一篇讲 Facebook 架构和国内对比的文章，感觉自己真是井底之蛙。头脑中一些架构方面的概念和 Status of the Art 的理念还相去甚远，迫切想要进一步了解一些先进知识。比如说，以前觉得 git flow 这个概念还挺不错的，实践了半年，发现 develop 分支完全是多余的；以前觉得每个项目分一个仓库方便管理，现在觉得 monorepo 似乎更好一点。另外就是对“互联网时代的 C 语言” Golang 有点想了解一下。

一年前休假的时候看了几眼 Golang，感觉还不错，但是想实际写点什么的时候发现 GOPATH 这个设计真是奇葩至极。而现在我的思想已经完全倒向 Monorepo 了，那么 GOPATH 也就看起来很可爱了，Golang 看起来也就很可爱了，也就决定再翻翻 Go 语言的书吧，以后说不定会写点儿什么呢。

忘了在哪里看过一句话：人的知识像一个网络，新学到的知识只有和已有的知识关联起来才能真正记得住、记得牢，否则的话像是一个孤岛的新知识很快就会被忘记了，于是就有了本文。

需要注意的是，本文并不是一个简单的语法对比，倘若只是语法的话，直接把代码一列其实就差不多了。除去语法之外，本文还在设计理念上做了一些对比。以下为目录。（没有链接的表示还没有写，敬请期待）

Ref

Python slice notation. https://stackoverflow.com/questions/509211/understanding-slice-notation/50929x
How to get type of go. https://stackoverflow.com/questions/20170275/how-to-find-a-type-of-an-object-in-go
Golang online repo. https://repl.it/languages/go
A tour of go. https://tour.golang.org/moretypes/6
golang vs python. http://govspy.peterbe.com/#lists
https://www.353.solutions/py2go/index.html

LeetCode 1236/1242 设计一个(多线程)爬虫解法

Posted on 2019年11月3日 by 逸飞

单线程题目 LeetCode-1236

具体题目就不说了，直接去 LeetCode 上看就好了。1236 要求使用单线程即可，考察的主要是图的遍历。只要注意到对于新发现的节点需要考虑是否已经访问过就好了。在实际生产中，肯定也是要用广度优先，深度优先基本就会陷进一个网站出不来了。

from urllib.parse import urlsplit

class Solution:
    def crawl(self, startUrl: str, htmlParser: 'HtmlParser') -> List[str]:
        domain = urlsplit(startUrl).netloc
        q = [startUrl]
        visited = set([startUrl])
        while q:
            newUrls = []
            for url in q:
                urls = htmlParser.getUrls(url)
                for newUrl in urls:
                    u = urlsplit(newUrl)
                    if u.netloc != domain:
                        continue
                    if newUrl in visited:
                        continue
                    visited.add(newUrl)
                    newUrls.append(newUrl)
            q = newUrls
        return list(visited)

多线程题目 LeetCode-1242

1242 题要求使用多线程来实现。在现实生活中，爬虫作为一个 IO 密集型的任务，使用多线程是一项必须的优化。

在上述的单线程版本中，我们使用了 visited 这个数组来存放已经访问过的节点，如果我们采用多线程的话，并且在每个线程中并发判断某个 URL 是否被访问过，那么势必需要给这个变量加一个锁。而我们知道，在多线程程序中，加锁往往造成性能损失最大，最容易引起潜在的 bug。那么有没有一种办法可以不用显式加锁呢？

其实也很简单，我们只要把需要把并发访问的部分放到一个线程里就好了。这个想法是最近阅读 The Go Programming Language 得到的启发。全部代码如下：

import threading
import queue
from urllib.parse import urlsplit

class Solution:
    def crawl(self, startUrl: str, htmlParser: 'HtmlParser') -> List[str]:
        domain = urlsplit(startUrl).netloc
        requestQueue = queue.Queue()
        resultQueue = queue.Queue()
        requestQueue.put(startUrl)
        for _ in range(5):
            t = threading.Thread(target=self._crawl, 
                args=(domain, htmlParser, requestQueue, resultQueue))
            t.daemon = True
            t.start()
        running = 1
        visited = set([startUrl])
        while running > 0:
            urls = resultQueue.get()
            for url in urls:
                if url in visited:
                    continue
                visited.add(url)
                requestQueue.put(url)
                running += 1
            running -= 1
        return list(visited)

    def _crawl(self, domain, htmlParser, requestQueue, resultQueue):
        while True:
            url = requestQueue.get()
            urls = htmlParser.getUrls(url)
            newUrls = []
            for url in urls:
                u = urlsplit(url)
                if u.netloc == domain:
                    newUrls.append(url)
            resultQueue.put(newUrls)

在上面的代码中，我们开启了 5 个线程并发请求，每个 worker 线程都做同样的事情：

从 requestQueue 中读取一个待访问的 url；
执行一个很耗时的网络请求：htmlParser.getUrls；
然后把获取到的新的 url 处理后放到 resultQueue 中。

而在主线程中：

从 resultQueue 中读取一个访问的结果
判断每个 URL 是否已经被访问过
并分发到 requestQueue 中。

我们可以看到在上述的过程中并没有显式使用锁（当然 queue 本身是带锁的）。原因就在于，我们把对于需要并发访问的结构限制在了一个线程中。

Netflix 公司文化学习笔记

Posted on 2019年10月26日2020年1月12日 by 逸飞

在头条的时候经常听一鸣说到“context, not control”，开始以为是他原创的，后来才知道原来是从 Netflix “偷”来的。不过当时其实自己并不能特别理解这其中的含义，直到后来在创业公司，偶然想到才感觉真是醍醐灌顶，于是特意找来完整的 Netflix 文化 PPT 学习了一下，并做了一写笔记。

file

Value are what we value

file
file

公司价值观体现在我们珍视什么。几乎每个公司都有一些听起来很高大上的价值观，然而这些金玉在外的话语是没有用的。

公司真正的价值观体现在公司奖励哪些行为，提升哪些人，以及解雇哪些人。
公司的价值观也体现在员工们珍视的行为和技能中。

High Performance

file
file
file
file
file
file

Freedom & Responsibility

file
file
file

Context, not Control (My favorite part :p )

file
file
file
file

Highly Aligned, Loosely Coupled

file

Pay Top of Market

file
file

Promotions and Development

file
file
file

What are some profitable one-person online businesses?

Posted on 2019年10月25日2020年1月21日 by 逸飞

Here are a list of examples I collected from Hacker News that claim to run a successful one-person online business. You could find the original threads at the end of this post.

Quiz Website

I run a popular Quiz website. I make around $6,000 per month from Google adsense. I work between 2-3 hours a week usually posting quiz links on my Pinterest page. My only expense is hosting which is around $20 per month (Digital Ocean). I have never advertised my website and it gets all the traffic from Pinterest Organically. Compare to my salary, I’m an IT Administrator in my day job and make $400 per month. I live in Ethiopia 🙂 I thought this inspires my fellow HN. Good day.

How do you manage to kickstart something like this? You mentioned you get your traffic organically via Pinterest, but there had to be something you did initially that set off that growth.

My website started five years ago, It didn’t get any traffic the first three years before one of my quiz went viral. Now I have around 70k followers on Pinterest.

This is important. I have seen this a lot. Persistence. Many people keep pushing,keep pushing even if there is no positive feedback loop for a long time. After a while, they beat time. Kudos.

Cursive-IDE

I develop and sell Cursive (https://cursive-ide.com), which has paid my bills nicely for a couple of years now. Currently I make more than I made in my last job at Google. I never thought I’d be able to make a living selling developer tools, much less into a niche market, but I’m constantly amazed by how well Cursive does.

The work is a mix of fun and boring slog, like most jobs I guess. A lot of my time is spent on support, both technical and sales, so when I work less I actually end up getting more frustrated because a higher percentage of the work is not as fun as writing new features. I’ve also had a bad year of having to work around IntelliJ bugs, but normally I like the actual development work a lot. I have friendly enthusiastic users who constantly make my day. It’s a pretty sweet gig, and being able to decide how I spend my time, and which bits of my time I spend working, is priceless.

I got started during a sabbatical from my last job, just building something that I wanted myself. It turns out that lots of other people wanted it too.

Product Pix

I own Product Pix (https://www.proproductpix.org). It removes the background from product photos, with the intended audience being mostly people who sell stuff online and need to set their background against a white background.

It makes $1300/month right now, up from $0 6 months ago. Living in the Bay Area, that would put me well below the poverty line if it were my sole source of income, so I’m not gonna call it "successful" just yet.

How I got started: I do machine learning, and I methodically searched for places where people buy a service transactionally on platforms like Fiverr and that I think can be automated away (or greatly automated with human reviewers in the loop) with state of the art machine learning models. There are hundreds or thousands of such opportunities that individuals can solve on their own.

I’ll be more comfortable giving sage advice once I’ve crossed the $10K/month threshold, but still I’d say a willingness to try a lot of shit out and get digging on stuff you have 0 familiarity with is mandatory. In this project I’ve had to learn javascript, frontend, photography, google ads campaign management etc.

Another tip I wish someone had told me is, build a pricing page from day one. The temptation to get some signal you’re useful to people will drive you to offer stuff for free, but that will end up getting you a lot of unwanted attention from people who will never ever pay.

Instagram Posting Service

My business is in my bio, don’t want to link it here. Pays about the same as my previous job at Microsoft did, but with a lot less involvement — I haven’t touched the main code in about a year now. I probably spend about two or three hours a week on customer support, that’s it, really. No marketing spend, all word-of-mouth and Google.

The idea came about when I wanted to post to Instagram, but the API didn’t allow it. So I spent about a week trying to automate the process using a phone, with screenshot OCR and a state machine. After a lot of messing around with it, I had a working prototype. Made a website, added a $5/month Stripe plan to see if people were willing to pay for it, sent it to a few friends, posted it on Twitter, and eventually, people signed up and tried it out. It worked, then it didn’t work, then I fixed it, then it worked again, this went on and on for a few weeks until it became quite useable.

About two months in, local offices of Toyota and Samsung signed up, and they loved it, money wasn’t an issue. That was the moment I realized it may be worth doing it properly.

It grew organically, and I bought lots and lots of Android phones, which are simple workers getting jobs off a queue, and host them in two locations roughly. Phones last for about two years, then I buy new ones (<$100 a phone). Each phone pays for itself in less than a month, server costs are less than $200 a month.

Facebook tried to sue me after I filed for a trademark, we figured it out (I rebranded). Been going steady ever since, but I consider it to be shut down by yet another Instagram move sooner or later. But I said that after 3 weeks of running it, and it’s been almost five years I think.

I made it a point to not use any private Instagram APIs, like all my competitors did — instead, I don’t emulate the Instagram app, I emulate the person tapping the phone, and use only the official app for it. I think that let me survive this long.

Updown.io

I run https://updown.io since 2012, a website monitoring service I created. I’m working about 5-10 hours per week on it. It makes about $6,000 per month and is still growing linearly. I also keep a full-time job alongside for now as an engineering manager. The key for me is to take time, make something useful, delight your clients, and don’t try to become uber or airbnb.

Sports App Business

I am mainly in sports apps. I think it is still possible to have succes. It requires a lot of patience. Don’t focus on the revenue part. And don’t try to build a new hype. Very slim chance you build the next angry birds. Instead try to build a product that is based on an already successful specific category/ product. Very important is that you understand your customer and genuinely try to make a product that is better than the competition. You should love your own product. The good thing is that bigger companies tend to destroy their own product with too many ads, notifications, non relevant features etc. Furthermore I believe it’s important that your product contents can be automated without too much manual work. After all you are the only person with only so much time. I know a guy who created a fitness diet app. He cooked and photographed more than a thousand meals. He wrote many articles. In the end he gave up. It took him 80 hours per week to maintain and update all the content. His app was making maybe 100 a month. I know another guy who created a successful formula 1 live app. He is using paid data feeds and scrapes a lot of additional data. Everything automated. Spends like 10 hours a week maintaining things. Makes about 100k a month. Similar story for a guy who created a popular weather app. In essence the only thing what they do is aggregation of data and present it in a relatively simple app. Also don’t spend too much time on analytics, seo and other optimizations. It may take 2 years before you get traction anyway. First the product then after (if it’s worth) the optimization. One concrete product where I think you can still have success is a baby monitor with 2 phones. Couple of good apps only. All premium priced. Not too difficult technically. I don’t have time for it, so go for it 🙂

PageFlow

I run https://pageflows.com and have been living off it full time for a little over a year.

The business makes a bit more than what I was earning a few years ago as a junior developer in London, so it’s not a huge amount of money, but it’s enough.
It’s a fairly boring business to run and not as predictable or sexy as some sort of micro saas, but it’s I’m happy with how things have been so far. Happy to answer any questions you have.

Most customers are indeed businesses. Great shout on some sort of team/business plan – it’s on my to-do list!

I’ve commended to a response below with the business model, but yeah I’ve just started trialling a freemium model yesterday so need to update the rest of the site with clearer pricing plans etc.

Until yesterday there was no freemium access, it was just paid up-front to access all the content. $39 per quarter or $99 per year.

Your use case is kinda where the idea came from, most product people do something similar. The hard part is adding enough relevant content on Page Flows for enough people!

Wakatime

Seven years ago I solo-started an automatic time tracker for programmers called WakaTime 1 and launched here on HN 2. Partly from listening to developers too much, I waited way too long (almost a year) before adding a paid plan, but now it generates more MRR than an SF developer salary not including stock options. Technically I make more from RSUs and stock from past startups as a regular employee, but if I wasn’t lucky with those then it would be my highest income stream.

1: https://wakatime.com/about
2: https://news.ycombinator.com/item?id=6046227

For anyone thinking it’s egregiously difficult to start a solo-project: You’re right, but if you stick with it your persistence will pay off. For solo-products, I think grit is the deciding factor between success and failure.

There were several stages of MVP. First usable version took a month and half to build and public launch with 2 IDEs supported was 2 and half months after starting to build.

May 3 2013 – Started development Flask website & Vim plugin (https://wakatime.com/blog/1-why-i-built-wakatime)
June 25 2013 – Finished Vim plugin and Website (https://github.com/wakatime/vim-wakatime/commit/4346a055e301…)
July 1 2013 – Started Sublime plugin (https://github.com/wakatime/sublime-wakatime/commit/b7fe36f8…)
July 15 2013 – Finished Sublime plugin and public launch (https://news.ycombinator.com/item?id=6046227)

Unfortunately I don’t have WakaTime data until after finishing the Vim plugin, but everything after that I can see how long the actual coding took by dogfooding.

ERP plugins

I have a one-person lifestyle business. I like it primarily because it gives me the flexibility to live anywhere in the world. I hated my old desk job and the idea of 2 weeks vacation every year.

I run a SaaS product that integrates with ERPs. I pretend to my customers that I have a team (so much so that I have multiple email addresses to people that don’t exist that actually just forward to me). One of our customers thinks they’re paying for a team of 6, but it’s actually just me.

My monthly billings last month was 73k USD. I am a tax resident of a tax haven although I do live 3-6 months at a time in a different country.

The only advice I’d give anyone looking to build a lifestyle business is to keep your ambitions and by extension- product feature set in check. I know several other people who operate like me, and the common thread is we have businesses that can easily take VC funds, hire, and expand. But for lifestyle priorities, we chose not to.

A lot of people I’ve met (particularly in Chiang Mai, Thailand) copy popular, common, and easy online businesses such as drop shipping, social media XYZ, or coding. Unless you live in a really low cost area, it’s not a good life. The key is have a very specific niche that can be scaled upwards if you want, but you always have the option not to. Those the ideas and businesses that seems to provide the ideal balance in lifestyle.

EDIT: The product came about at my last job where I built it to make my own job easier. Essentially it did 95% of what job which at the time enabled me to be the "best performer" while not actually working that hard.

Flowx.io

I develop Flowx [https://www.flowx.io], an Android weather app. It makes around $2,500 USD/month with about $500/month in costs excluding my time. It covers about 60% of my total costs including my time which is 40+ hours a week. I cover my remaining costs through contract work. This might not seem like a success but the business allowed us to move to the Rarotonga, Cook Islands from Auckland, New Zealand. Lifestyle-wise and building-a-business-wise, I think it’s a success.

Just an added note. I started Flowx as a side-project in 2012. In 2016, we moved to Rarotonga and we decided then to try to grow it into a business. It was making ~$100/month at that stage. Since then, it has grown it to $2500/month through added pro features and a better subscription prices.

Stay focused

I sell freemium software that blocks distractions on your computer so that you can focus on doing work. Unlike my competitors, it’s a one-time payment business model.
The idea for my product first came to me when a friend in university had trouble staying focused on writing papers. He was constantly playing World of Warcraft and needed a way to temporarily block himself from playing the game. So I quickly made a little VB.NET app and service that would watch for the game executable and kill the process if it starts. It did the job well enough and he ended up graduating 🙂

At that point, some other students approached me and asked for my little app to help them study. That’s when, half-way through university (2010), I made a website for my app and had it available for free. I continued to maintain it and over 4 years, added more features including: blocking websites, adding breaks, scheduling, and passwords.

In 2014, I split the product into a free and paid tier. It wasn’t an easy decision, but I was spending a lot of time on it by this point and customer support was also starting to take a serious hit on my personal time. In about two years (2016), I was making more money from the paid product than my well paying government day job. So, I decided to quit my job and work on my business full time.

Although I felt it was risky, the alternative was passing up an opportunity many people dreamed of having. I never planned to start a business in the first place and I kind of felt/still feel imposter syndrome. For now, I’m just enjoying my new found freedom and continue to be thankful for my new job. I’m going to keep it a lifestyle business for now, but I wouldn’t be opposed to selling it as my exit plan.

I’ve spent (effectively) $0 in advertising since developing it and I’d say my customers come from organic search, external links, and word-of-mouth.

browserless.io

I run a headless browser service called browserless.io. Got started due to lack of a comparable service, and all others seemed more geared for testing.

It’s been around two years now, and makes more than any prior engineering job I’ve ever had. You do have a lot of other stresses you might not otherwise have, but you’ll also work a lot less than at a traditional job!

I’m working on a few interviews for some sites, which go more into the details, and will post here when they’re done.

EDIT: feel free to comment here on anything or email me at joel at browserless dot io

Website builder

I created a SaaS website builder for a small niche market. I’ve been running it for about 8 years. I gross a bit over $14,000 per month with about $500 in expenses for servers and third-party APIs. I work 1-2 hours per week answering customer support emails. Basically, I automated away my old job as a web designer 🙂

The smartest decision I made was targeting a small niche market that larger businesses wouldn’t bother with. I often get kind emails from customers thanking me for helping their industry. I kept things simple, didn’t add features unless I really believed customers needed them, and didn’t try to generalize the solution. I think those are the main reasons why the product worked.

By far the hardest part was/is marketing. I’m still bad at it. I’ve tried may things. Most failed or were too hard to sustain. Some succeeded, like Facebook ads, but those successes were often hard to recreate. At this point it’s mostly word of mouth.

Working alone can be psychologically challenging. When I have a problem, there’s no one to help because no one else knows how the platform works. With no one to bounce ideas off of, it’s easy to get stuck in a rut going round and round the same set of possible solutions. And I really have to monitor myself to ensure that I don’t get too isolated. This was an issue in the early years, but now I have a routine that gets me up and out and into the world every day. I would strongly advise anyone considering the solo route to carefully consider the social and mental health aspects of working alone.

I feel very grateful to my former self for doing the hard work that pays my bills today. And I’m tremendously grateful for open source tools and resources like Stack Overflow without which I would never have made it this far alone.

meme tshirt

I ran a Shopify site selling meme shirts for 3 years.

You might recognize classics such as "Legalize 4Loko 2020" and "BREAD" as featured in Elle magazine.

All on-demand printing. Order goes through Shopify’s API to the supply center, order gets fulfilled, shipped. No inventory. Kinda pricey, but zero maintenance. Set and forget.

Find the most extremely dank and niche memes possible so you hit the little nugget inside of someone’s brain that makes them want to spend $15-30 on a t-shirt.

A good print would net me somewhere like 300 orders a month. A sweatshirt could go for $50-60. You have options.

ML golf predict

I made https://www.golfforecast.co.uk – an ML algorithm to predict golf.

After 5 years it’s making enough from subscriptions for me to live off (3K gbp/mo). The algorithm is always a work in progress but it’s seeing consistent returns now so I’m making money from that too 🙂 plus it makes golf a lot more entertaining.

IPINFO.IO

I started https://ipinfo.io as a side project, and then ran it fulltime as a one-person SaaS app for over a year. We’re now a team of 8, profitable, and growing quickly. We’re still 100% bootstrapped, and I have zero plans to raise any outside funding.

We started with a simple IP geolocation API, which now handles over 20 billion API requests per month. We’ve added new data to that service, such as IP type classification (hosting, isp, or business, and soon education too), IP to company, and carrier detection. And we’ve also launched some other products, like hosted domain API (all domains hosted on an IP, sometimes called reverse IP), IP ranges belonging to an organization, and an ASN API. We’ve got a lot in the pipeline too, including some domain related offerings (see https://host.io for an early preview).

So it’s definitely possible 🙂 What sort of SaaS product are you thinking of launching? Would be happy to chat! Shoot me an email at ben@ipinfo.io

NANAGRAM

I’m working on NanaGram (https://nanagram.co) solo and bootstrapped. Although I’m not making a full-time income yet, it’s generating a profit. It’s mostly automated.

NanaGram is the 3rd greatest generator of happiness and fulfillment in my life (after my wife and my dog). I get a constant stream of good vibes from customers, most recently voicemails from grandmothers! (https://nanagram.co/blog/feedback-by-vm)

Good luck 🙂

PhantomJSCloud

I run https://PhantomJsCloud.com
I started it as a free MVP about 2 years ago while in Thailand, and given that I was attracting a slow but steady stream of users I decided to build out a commercial v1 from it.

The freemium SaaS went live in March and it’s growing monthly. If I still lived in Thailand I would consider it very successful, but I am in the Seattle area now so it’s ramen profitable.

The biggest surprise I got was how slow organic growth takes. Every month I gain more users + MRR but discovery seems to be the biggest problem. I tried Google Adwords in June but Google decided to cost me upwards of $5/click for basic keyword targeting so gave that up. I tried Adwords again in November and now google thinks I’m more relevant, so I pay starting at $0.20/click for the same keywords that cost $5/click 6 months previous. I am currently doing experiments to see if the acquisition cost justifies that spend.

From a effort perspective, the SaaS api+backend itself was about 50% of the effort. The subscription service + user dashboard was another 50%.

From a skills perspective, I think doing a SaaS as a solo founder is only practical if you have extremely broad skillsets: Business management, UX, full-stack webdev, devops, sales, marketing, support. Thankfully I have some experience in all those (except sales) so I was able to either do or fake everything required. If you don’t have all those skills, you are going to be increasingly reliant on luck, which isn’t a winning strategy.

I solicit users to email whenever they have a question/comment/issue and reply to everything. Overall I think I have provided email support to aprox 50% of my paying customers, and maybe half of the support was provided before they decided to pay, so it is very important 🙂

it’s actually a really great way to understand your customer’s needs, and your products actual (in the eyes of the user) deficiencies. I also use uservoice to help the highly-desired features/requests to "bubble up" but if a customer asks for something and it’s an easy enhancement I go ahead and implement it. Likewise if the same problem is annoying a bunch of people, I need to either document a workaround or make it easier.

Yes, "inbound marketing" (a blog) is probably the biggest accelerator to growth I can (and should) do. I’m holding off for now though, as I need to make the product more friendly to business users first. Right now PhantomJsCloud is focused on developers, so I need to make some non-dev friendly tooling first. That’s my excuse at least.
Regarding StackOverflow, yes, that’s actually how I validated the free MVP (answering SO questions and if my product might be beneficial, providing a link to my product) but generally those traffic sources don’t seem to scale very well past MVP validation. I haven’t tried Quora though, I will add that to my todo list 🙂

SingalBox

I’m running https://SignalBox.ai alone, I wrote all of the software and am working on partnering and sales right now.
Previously I have 2 other startups, one was media monitoring and one was forex.

The media monitoring is B2B only. The forex trading is automated and run from my home research cluster.

Both are generating enough revenue to live off (media monitoring 120k forex, 60-80k)

I guess they fit the definition of solo founder and online, but they have no public facing websites (except SignalBox)

EDIT: I also run a slack group for Solo Founders, If you would like an invite, please email me

Sales are a big struggle to me. Where did you find this partnerships?

Network. Go to the meetups.
Don’t rely on serendipity, we can do better than that. Use your programming skills.

Pull the meetup list, get all of their twitter profiles, search everyones last 1000 tweets for topics you are interested in. Pull all of their code on github. Push it through the profiler and find the talent.

Mirror github if you have to. Pull the whole darn thing, it’s only a couple of hundred gigs (if you dont pull the code) Profile everyone based on their stars, contributions, watchers and pull requests.

How many other meetups do they go to? What’s their history like on other forums?

Put the pics of these people on your phone, and then go and find them at the meetup. Pull their customer lists / testimonials and any other publicly available data.

Look at their company DNS records. Pull their company filings if they’re available. Know their revenue, know their customers. Who’s making the decisions at this company? Who is signing the cheques?

Scientia potentia est

dropshipping guitars on shopify

I run an ecommerce store from Shopify which fulfills the orders by drop-shipping through AliExpress.
This is definitely doable for one person, and it isn’t technically challenging for a software developer–but the hardest part (at least for me) is marketing, creating content, advertising, and so on.

Actually running a Shopify store and fulfilling by drop-shipping is simple. I would definitely recommend that as a good place to start, one person can do it.

There are a few good ways, but it really helps if you know the products well. For me, my site sells guitar parts and DIY kits. And I’ve been playing guitar since I was about 10 years old, so that helps a ton.
I had a few other stores before this that didn’t sell well at all, and I have to say that’s because I just didn’t know the products, or what the end users really wanted/needed/cared about.

Great ways to pick products: – Terapeak (http://www.terapeak.com/), but this is paid – eBay completed listings – Or most simple (and what I use) — once you know your products, search AliExpress and sort by "best-selling". That’s my go-to.

Feel free to check my store for ideas (or if you want to buy something!). URL is: http://modshop.guitars/

pinpoard

I run Pinboard, $257K in gross revenue for 2016. A ton of money for one person, not quite enough for two people.

Do you have a writeup on your marketing?

My marketing is, I spend all my time talking smack on Twitter.

bugmuncher

I run BugMuncher (https://www.bugmuncher.com), it started as a side-project 5 years ago, then in September 2015 I packed in freelancing to focus on BugMuncher full time.

As of November 2016 BugMuncher reached profitability – ie: it’s my sole source of income, and covers all of my living expenses.

price comparison

I run https://www.fortsu.es (also https://www.fortsu.co.uk, https://www.fortsu.de and https://www.fortsu.com) a price comparison website for running shoes. Original one is focused on spanish market while expanding into interesting ones.

It started as side project some years ago when I wanted to buy running shoes online and it has been improved over the time. To-Do list never ends 😉

At the beginning it was basically word of mouth and niche related forums on the internet.
Then I started reading about SEO and advertising. Organic search more or less work but I got almost no traffic from a couple of banners on related pages during few months. I didn’t try advertising networks like AdWords.

I don’t see big brands as my competition. I have partnered with some but I don’t think they sell much on the internet (typically higher price tags) compared to full equipped city centre stores.

iOS app

I started selling macOS (and now iOS) software on my own website back in 2007. https://clickontyler.com My original goal was to earn enough money to refinish the hardwood floors in my house. Since then, however, it’s taken on a life of its own and become a suite of three main products. It enables me to live comfortably in the Nashville suburbs.

webhosting reviews

1 man startup – http://reviewsignal.com/webhosting/compare I do web hosting reviews. Not the scummy pay-for-placement stuff you see, but an actual review site. It tracks what people are saying about hosting companies on Twitter and publishes the results.
The story is told a bit here http://techcrunch.com/2012/09/25/web-hosting-reviews-are-a-c… I was just tired after 10 years of still relying exclusively on my experience and the experiences of people I knew. Figured there must be a better way and I had been working with Twitter data for thesis and saw this opportunity.

selling online

I started selling online, total sales so far over $300k. Multiple sources, some retail, some wholesale.
What I’ve learned:

Not all rules matter. A large part of my business is stretching certain rules, either from the marketplace, or from the source (e.g. a store that doesn’t allow resale). That said, you can’t get away with breaking rules unless you have a very good understanding of why the rule exists, who’s motivated to uphold it, and generally what the risks are. Don’t screw over customers.
There’s a lot more to be made by taking risks than there is to be lost. I’ve easily lost over $1k multiple times in various ways, but when I "win" it’s to the tune of 10 or 30 times that. Take smart risks, only where the realistic upside justifies it.
Be willing to pay for information. There are courses out there in almost any topic. Personally I’ve largely carved my own path and paid very little , but I’d still recommend courses for others. Also read a lot of whatever free information is out there, and network with people who have more experience.
Don’t do too many things at once. It will kill you. I’ve been full time in college and it’s extremely tough to balance everything. Delegate as soon as you can afford to, anything others can do that doesn’t take a lot of brains pay people to do.
Don’t be afraid to scale, but do it slowly. My first purchase of over 10k was 6 months after I started, iirc.

(Several of these are probably specific to this kind of business, may not be generally applicable. Startups have a much different road where profitability isn’t the most important at first.)

selling open source software to government

I was working for Automattic after an acqui-hire thing. After a year there, I found that I missed working in security. I found a full-scope penetration testing gig three blocks from my apartment.

In my spare time, I started to tinker with a few ideas and released them as an open source project. Said project saw a lot of interest within the hacker community very quickly. I didn’t expect this. Folks formed an opinion on it pretty quickly. Some people hate it. Others love it. Of those who know it, very few are in-between.

I left my pen testing job with a decent amount of money saved up. I didn’t know exactly what I would go and do afterwards. I spent some time tinkering with Android, just for giggles.

I was very reluctant to start a business that used my "successful?" open source project. Partially because it leverages another open source project owned by another company.

I was at a conference in 2011 and someone from a US government agency asked if I was selling anything. I said no. He said that was too bad, because he had end of year money, and he liked my open source stuff. It was then that I decided to look at expanding my open source kit into a commercial product.

April will mark the two year anniversary of my first customer. My customers are well known organizations and they trust my software to assess how well they protect their networks. I’m constantly in awe of this.

website counter

https://www.improvely.com and https://www.w3counter.com

Five figures a month, just me, I’ve written about my solo business a couple times in other Ask HN threads. Ten years ago (almost to the day), in my college dorm, I was looking at the Webalizer web stats report my web host provided for my blog, and thought "I could do something much cooler than this". So I did. I had built a few educational sites and threw some ads on them for a couple years before that, but W3Counter was the first service I actually charged a subscription for, and now I make a living building and selling this stuff.

vintage computer hardware

I don’t know exactly how you define "successful online business," but I am currently a university student making $500 – $2000 a month at about 5 to 10 hours a week.
Basically, there is a market for vintage computer hardware, so I post some adds offering to take away old office items they can’t just throw away. Such as old keyboards, terminals, etc. and they pay me a nominal fee ($1 – $5 per item depending) to rid them of their "trash". I then resell those items after cleaning them up a bit for extremely high profit margins $35 – $120 for 20 minutes of work (since I was payed to take away the trash).

One of the things I did was sold Model M keyboards which I made USB compatible: http://austingwalters.com/keyboards/

Another way I make money is by tutoring or helping out with programming, I use to help out local people, but I have since switched over to Google Helpouts. Usually, it’s just explaining some algorithms and writing some C code. Pretty easy, no real upkeep, and I can set what ever hours I want.

pingrow

Just launched Pinegrow Web Designer (http://pinegrow.com) two months ago. The company is actually run by my wife and me, but I do all the work with Pinegrow while she is taking care of our other projects.
Pinegrow has been paying most of our bills since launch and I have a lot of expansions in the pipeline: full support for Foundation alongside Bootstrap, developer edition that’ll work with templates, a similar app for designing emails…

cramfighter

I run a small business called Cram Fighter (http://cramfighter.com) that is targeted at students (mostly medical) that are preparing for standardized exams. I got the idea after watching my wife preparing for her board exams and it seemed like a perfect little project to learn iOS programming. Initially my goal was to do earn maybe $5k annually, but now I’m on track to surpass my salary as senior developer by next year.
You’ll find a lot of one-person businesses targeting tiny, but profitable, niches like mine. What’s great about it is that often when you find a tiny opportunity, it opens up a lot of other problems that need solving that you would never find otherwise. It’s also a great way to learn the skills of running a business in a relatively stress-free way (at least compared to running a startup).

The only downside is if you’re anything like me, you’ll get antsy working on small projects and yearn to tackle bigger, more ambitious problems. Sometimes 1-person companies have the potential for turning into a company with startup-like growth, sometimes not. I’m still trying to figure out how far I can take my company.

office snapshots

I run http://officesnapshots.com which publishes photos of office design projects from around the world.
I started it in 2007 as a gin side project to teaching history. I’m no longer teaching and it is the majority of my income.

laptop battery meter

I sell a laptop battery meter (http://batterybarpro.com). It’s not income replacing; it makes about $1,000 per month, but it’s been crucial in saving enough for down payments one two houses.
I’ve tried to get the revenue numbers up, but I’ve never been able to break a $2,000 month.

robots everywhere

http://www.robots-everywhere.com I used to employ two people, but I automated them away. I am successful in the sense that I have clear title to my home at age 33, if that counts.

excel version control

I run https://www.spreadgit.com, a hosted version control system for Excel. Doing this solo and full time. It’s been a hell of a ride so far but I love it.

write a book

I wrote a book1 that generates about $2k of revenue per month. Not quite your definition of success, but it’s given me a taste. I’m now in the beta testing process for my next thing2.

smart shooter

I develop and sell Smart Shooter.
http://kuvacode.com

Its a traditional desktop app (windows, mac), but only sold online via our own website or the mac app store. I created it about 4 years ago, and work on it solely in my spare time. In fact I’m employed full time at a major tech company but this I keep separate.

To claim its profitable is a bit misleading, because of cause the major cost in developing such software is my own time. I’ve incorporated as a limited company here in Finland but do not pay myself a salary, so the only costs to the business are web hosting and occasional hardware purchases (computers, cameras).

I started this as a project for personal interest; at the time I was working as a software engineer developing financial trading software. Smart Shooter was a good way to develop something that covered both my interests in graphics programming and digital photography, to alleviate the borebom from my day job.

So for me its been successful, its still an pleasureable hobby, allows me an excuse to play around with the latest cameras, and brings in some pocket money. It doesn’t generate enough revenue that I could quit my main job, but the possibilities could be there if situations change.

payment gateway

I am running a complete payment gateway that supports VISA and MasterCard and mobile payments by SMS.
The name of the service is: https://www.bizify.me

For an introduction to the service: https://www.bizify.me/hacker-news/

asterisk consulting

I sell Asterisk reporting sw for win @samreports.com. It makes about $1000 a month in revenues. I also work as iOS developer for the man. I have a free iOS app on the AppStore (HRTecaj), soon to be commercial, when I add ATMs. I was Asterisk integrator, and learned a lot about the system, made software to present call reports in customisable and pleasant way. SAMReports has been selling, consistently, for 4 years. I made a few updates, but now I’m working on a major update.

sell books

I run http://lsathacks.com, and have a related book series
I sell e-books on my site and through affiliates, and sell print books on amazon. All told I make around $3,000 a month in passive revenues. I also make $4000-$5000 more in tutoring revenues.

However, the site is fairly new (I just sold the books through affiliates/print previously). As I grow the site I expect I may be able to get over $10,000 per month passive.

The LSAT is an admission exam for American and Canadian law schools. My materials/lessons teach people how to do better on it.

radio community

I own and operate http://www.radioreference.com and http://www.broadcastify.com. I do all the development, business management, and support.
I have a team of community volunteers that do a lot of day to day moderation and member management.

I got started simply building a set of community resources for the radio communications and hobbyist market.

We’re very profitable and these businesses provide the majority of my family’s income.

electricain calculator pro

I developed the Electrician Calculator Pro, a National Electrical Code compliant calculator for engineers, electricians, lighting designers, etc:
http://www.electriciancalculator.com

I first created the Android version about 3 years ago, then the iOS version about 1 year ago. It currently makes just enough to cover some bills, although I believe it has a greater potential. I’m currently looking for ways to make this a recurring revenue stream instead of a one time payment gig.

vlad studio – wallpapers

I’ve been running http://www.vladstudio.com (where I publish my wallpapers and other stuff) for several years, and for quite some time, it was my primary source of income. Unusual, because my premium accounts are not really a "product", but just a way to "like" or "donate".

m3u editor

Running a profitable SAAS has been my dream from the moment I wrote my first line of code.
Here on HN and IndieHackers I’ve always looked up to the people who pay their bills with recurring revenue from their tools.

I’ve tried, many times, to do the same, without much success. A couple of rather successful HN pitches, but none of my projects ever even paid me a beer (let alone my rent).

Until this month! Last year I built myself and my girlfriend a tool. Even though I did build it for other people to use it, I had never thought someone actually would. Long story short, half a year later I provide my service to more than 5000 (fully organic) users.

This month is the first month in which revenue is high enough to pay my rent with it. Disclaimer: I share my rent with my girlfriend, but it does sound cool to say.

Looking back at the proces, it does match with a lot of other success stories I read over the years in the HN community. The main lesson which I can now confirm: build something that scratches your own itch.

So… Thanks you guys, for keeping me motivated and inspired.

Gardening app

My only successful product to this date is an app I built because my wife asked me to. It is in an non-technical domain which I knew nothing about. I thought it was rather non-promising, but, since it was a pet-peeve of hers, I gave it a try.

It was an awesome (and very bonding) experience – she explained me the problem(s), and I tried to simplify and structure it (didn’t think gardening could be so complicated). Both of us were in their respective element, and from back and forth an app was forged.