知行编程网知行编程网  2022-07-07 11:00 知行编程网 隐藏边栏 |   抢沙发  3 
文章评分 0 次,平均分 0.0

没有什么内存问题,是一行Python代码解决不了的

来自 | 大数据文摘

编译 | Javen、胡笳、云舟
没有什么内存问题,是一行Python代码解决不了的


内存不足是项目开发过程中经常碰到的问题,我和我的团队在之前的一个项目中也遇到了这个问题,我们的项目需要存储和处理一个相当大的动态列表,测试人员经常向我抱怨内存不足。但是最终,我们通过添加一行简单的代码解决了这个问题。

结果如图所示:
没有什么内存问题,是一行Python代码解决不了的

我将在下面解释它的工作原理。

举一个简单的“learning”示例 - 创建一个DataItem类,在其中定义一些个人信息属性,例如姓名,年龄和地址。
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;"><span style="color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">class</span> <span style="line-height: inherit;color: rgb(165, 218, 45);overflow-wrap: inherit !important;word-break: inherit !important;">DataItem</span><span style="line-height: inherit;color: rgb(255, 152, 35);overflow-wrap: inherit !important;word-break: inherit !important;">(object)</span>:</span><br  />   <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"><span style="line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">def</span> <span style="line-height: inherit;color: rgb(165, 218, 45);overflow-wrap: inherit !important;word-break: inherit !important;">__init__</span><span style="line-height: inherit;color: rgb(255, 152, 35);overflow-wrap: inherit !important;word-break: inherit !important;">(self, name, age, address)</span>:</span><br  />       self.name = name<br  />       self.age = age<br  />       self.address = address</span></section>

   小测试:这样一个对象会占用多少内存?

首先让我们尝试下面这种测试方案:
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;">d1 = DataItem(<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"Alex"</span>, <span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">42</span>, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"-"</span>)<br  /><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">print</span> (<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"sys.getsizeof(d1):"</span>, sys.getsizeof(d1))</span></section>

答案是56字节。看起来比较小,结果令人满意。

但是,让我们检查另一个数据多一些的对象:
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;">d2 = DataItem(<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"Boris"</span>, <span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">24</span>, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"In the middle of nowhere"</span>)<br  /><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">print</span> (<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"sys.getsizeof(d2):"</span>, sys.getsizeof(d2))<br  /></span></section>

答案仍然是56。这让我们明白这个结果并不完全正确。

我们的直觉是对的,这个问题不是那么简单。Python是一种非常灵活的语言,具有动态类型,它在工作时存储了许多额外的数据。这些额外的数据本身就占了很多内存。

例如,sys.getsizeof(“ ”)返回33,没错,每个空行就多达33字节!并且sys.getsizeof(1)将为此数字返回24-24个字节(我建议C程序员们现在点击结束阅读,以免对Python的美丽失去信心)。

对于更复杂的元素,例如字典,sys.getsizeof(dict())返回272个字节,这还只是一个空字典。举例到此为止,但事实已经很清楚了,何况RAM的制造商也需要出售他们的芯片。

现在,让我们回到回到我们的DataItem类和“小测试”问题。

   这个类到底占多少内存?

首先,我们将以较低级别输出该类的全部内容:
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;"><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"><span style="line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">def</span> <span style="line-height: inherit;color: rgb(165, 218, 45);overflow-wrap: inherit !important;word-break: inherit !important;">dump</span><span style="line-height: inherit;color: rgb(255, 152, 35);overflow-wrap: inherit !important;word-break: inherit !important;">(obj)</span>:</span><br  />    <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">for</span> attr <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">in</span> dir(obj):<br  />        print(<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"  obj.%s = %r"</span> % (attr, getattr(obj, attr)))</span></section>

这个函数将显示隐藏在“隐身衣”下的内容,以便所有Python函数(类型,继承和其他包)都可以运行。

结果令人印象深刻:
没有什么内存问题,是一行Python代码解决不了的

   它总共占用多少内存呢?

在GitHub上,有一个函数可以计算实际大小,通过递归调用所有对象的getsizeof实现。
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;"><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"><span style="line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">def</span> <span style="line-height: inherit;color: rgb(165, 218, 45);overflow-wrap: inherit !important;word-break: inherit !important;">get_size</span><span style="line-height: inherit;color: rgb(255, 152, 35);overflow-wrap: inherit !important;word-break: inherit !important;">(obj, seen=None)</span>:</span><br  />   <span style="line-height: inherit;color: rgb(128, 128, 128);overflow-wrap: inherit !important;word-break: inherit !important;"># From https://goshippo.com/blog/measure-real-size-any-python-object/</span><br  />   <span style="line-height: inherit;color: rgb(128, 128, 128);overflow-wrap: inherit !important;word-break: inherit !important;"># Recursively finds size of objects</span><br  />   size = sys.getsizeof(obj)<br  />   <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">if</span> seen <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">is</span> <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">None</span>:<br  />       seen = set()<br  />   obj_id = id(obj)<br  />   <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">if</span> obj_id <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">in</span> seen:<br  />       <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">return</span> <span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">0</span><br  /><br  /><span style="line-height: inherit;color: rgb(128, 128, 128);overflow-wrap: inherit !important;word-break: inherit !important;"># Important mark as seen *before* entering recursion to gracefully handle</span><br  />   <span style="line-height: inherit;color: rgb(128, 128, 128);overflow-wrap: inherit !important;word-break: inherit !important;"># self-referential objects</span><br  />   seen.add(obj_id)<br  />   <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">if</span> isinstance(obj, dict):<br  />     size += sum([get_size(v, seen) <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">for</span> v <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">in</span> obj.values()])<br  />     size += sum([get_size(k, seen) <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">for</span> k <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">in</span> obj.keys()])<br  />   <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">elif</span> hasattr(obj, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">'__dict__'</span>):<br  />     size += get_size(obj.__dict__, seen)<br  />   <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">elif</span> hasattr(obj, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">'__iter__'</span>) <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">and</span> <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">not</span> isinstance(obj, (str, bytes, bytearray)):<br  />     size += sum([get_size(i, seen) <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">for</span> i <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">in</span> obj])<br  />   <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">return</span> size<br  /><br  /><br  /><span style="line-height: inherit;color: rgb(128, 128, 128);overflow-wrap: inherit !important;word-break: inherit !important;">#让我们试一下:</span><br  /><br  />d1 = DataItem(<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"Alex"</span>, <span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">42</span>, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"-"</span>)<br  /><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">print</span> (<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"get_size(d1):"</span>, get_size(d1))<br  /><br  />d2 = DataItem(<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"Boris"</span>, <span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">24</span>, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"In the middle of nowhere"</span>)<br  /><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">print</span> (<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"get_size(d2):"</span>, get_size(d2))<br  /></span></section>


我们分别得到460和484字节,这似乎更接近事实。

使用这个函数,我们可以进行一系列实验。例如,我想知道如果DataItem放在列表中,数据将占用多少空间。

get_size([d1])函数返回532个字节,显然,这些是“原本的”460+一些额外开销。但是get_size([d1,d2])返回863个字节—小于460+484。get_size([d1,d2,d1])的结果更加有趣,它产生了871个字节,只是稍微多了一点,这说明Python很聪明,不会再为同一个对象分配内存。

现在我们来看问题的第二部分。

   是否有可能减少内存消耗?

答案是肯定的。Python是一个解释器,我们可以随时扩展我们的类,例如,添加一个新字段:
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;">d1 = DataItem(<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"Alex"</span>, <span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">42</span>, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"-"</span>)<br  /><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">print</span> (<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"get_size(d1):"</span>, get_size(d1))<br  /><br  />d1.weight = <span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">66</span><br  /><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">print</span> (<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"get_size(d1):"</span>, get_size(d1))</span></section>

这是一个很棒的特点,但是如果我们不需要这个功能,我们可以强制解释器使用__slots__指令来指定类属性列表:
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;"><span style="color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">class</span> <span style="line-height: inherit;color: rgb(165, 218, 45);overflow-wrap: inherit !important;word-break: inherit !important;">DataItem</span><span style="line-height: inherit;color: rgb(255, 152, 35);overflow-wrap: inherit !important;word-break: inherit !important;">(object)</span>:</span><br  />   __slots__ = [<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">'name'</span>, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">'age'</span>, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">'address'</span>]<br  />   <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"><span style="line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">def</span> <span style="line-height: inherit;color: rgb(165, 218, 45);overflow-wrap: inherit !important;word-break: inherit !important;">__init__</span><span style="line-height: inherit;color: rgb(255, 152, 35);overflow-wrap: inherit !important;word-break: inherit !important;">(self, name, age, address)</span>:</span><br  />       self.name = name<br  />       self.age = age<br  />       self.address = address</span></section>

更多信息可以参考文档中的“__dict__和__weakref__的部分。使用__dict__所节省的空间可能会很大”。

我们尝试后发现:get_size(d1)返回的是64字节,对比460直接,减少约7倍。作为奖励,对象的创建速度提高了约20%(请参阅文章的第一个屏幕截图)。

真正使用如此大的内存增益不会导致其他开销成本。只需添加元素即可创建100,000个数组,并查看内存消耗:
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;">data = []<br  /><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">for</span> p <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">in</span> range(<span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">100000</span>):<br  />   data.append(DataItem(<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"Alex"</span>, <span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">42</span>, <span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"middle of nowhere"</span>))<br  /><br  />snapshot = tracemalloc.take_snapshot()<br  />top_stats = snapshot.statistics(<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">'lineno'</span>)<br  />total = sum(stat.size <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">for</span> stat <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">in</span> top_stats)<br  />print(<span style="line-height: inherit;color: rgb(238, 220, 112);overflow-wrap: inherit !important;word-break: inherit !important;">"Total allocated size: %.1f MB"</span> % (total / (<span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">1024</span>*<span style="line-height: inherit;color: rgb(174, 135, 250);overflow-wrap: inherit !important;word-break: inherit !important;">1024</span>)))</span></section>

在没有__slots__的情况结果为16.8MB,而使用__slots__时为6.9MB。当然不是7倍,但考虑到代码变化很小,它的表现依然出色。

现在讨论一下这种方式的缺点。激活__slots__会禁止创建其他所有元素,包括__dict__,这意味着,例如,下面这种将结构转换为json的代码将不起作用:
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;"><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"><span style="line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">def</span> <span style="line-height: inherit;color: rgb(165, 218, 45);overflow-wrap: inherit !important;word-break: inherit !important;">toJSON</span><span style="line-height: inherit;color: rgb(255, 152, 35);overflow-wrap: inherit !important;word-break: inherit !important;">(self)</span>:</span><br  />    <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">return</span> json.dumps(self.__dict__)</span></section>

但这也很容易搞定,可以通过编程方式生成你的dict,遍历循环中的所有元素:
<section style="line-height: 18px;font-size: 14px;letter-spacing: 0px;font-family: Consolas, Inconsolata, Courier, monospace;border-radius: 0px;color: rgb(169, 183, 198);background: rgb(40, 43, 46);padding: 0.5em;margin-left: 8px;margin-right: 8px;overflow-wrap: normal !important;word-break: normal !important;overflow: auto !important;display: -webkit-box !important;"><span style="font-size: 16px;"><span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;"><span style="line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">def</span> <span style="line-height: inherit;color: rgb(165, 218, 45);overflow-wrap: inherit !important;word-break: inherit !important;">toJSON</span><span style="line-height: inherit;color: rgb(255, 152, 35);overflow-wrap: inherit !important;word-break: inherit !important;">(self)</span>:</span><br  />       data = dict()<br  />       <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">for</span> var <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">in</span> self.__slots__:<br  />           data[var] = getattr(self, var)<br  />       <span style="line-height: inherit;color: rgb(248, 35, 117);overflow-wrap: inherit !important;word-break: inherit !important;">return</span> json.dumps(data)</span></section>

向类中动态添加新变量也是不可能的,但在我们的项目里,这不是必需的。

下面是最后一个小测试。来看看整个程序需要多少内存。在程序末尾添加一个无限循环,使其持续运行,并查看Windows任务管理器中的内存消耗。

   没有__slots__时

没有什么内存问题,是一行Python代码解决不了的

69Mb变成27Mb......好吧,毕竟我们节省了内存。对于只添加一行代码的结果来说已经很好了。

注意:tracemalloc调试库使用了大量额外的内存。显然,它为每个创建的对象添加了额外的元素。如果你将其关闭,总内存消耗将会少得多,截图显示了2个选项:

没有什么内存问题,是一行Python代码解决不了的

   如何节省更多的内存?

可以使用numpy库,它允许你以C风格创建结构,但在这个的项目中,它需要更深入地改进代码,所以对我来说第一种方法就足够了。

奇怪的是,__slots__的使用从未在Habré上详细分析过,我希望这篇文章能够填补这一空白。

   结论

这篇文章看起来似乎是反Python的广告,但它根本不是。Python是非常可靠的(为了“删除”Python中的程序,你必须非常努力),这是一种易于阅读和方便编写的语言。在许多情况下,这些优点远胜过缺点,但如果你需要性能和效率的最大化,你可以使用numpy库像C++一样编写代码,它可以非常快速有效地处理数据。

最后,祝你编程愉快!
相关报道: 
https://medium.com/@alexmaisiura/python-how-to-reduce-memory-consumption-by-half-by-adding-just-one-line-of-code-56be6443d524
<pre style="font-size: 16px;letter-spacing: 0.544px;background-color: rgb(255, 255, 255);text-align: left;"><section data-brushtype="text" style="padding-right: 0em;padding-left: 0em;white-space: normal;max-width: 100%;letter-spacing: 0.544px;color: rgb(62, 62, 62);font-family: "Helvetica Neue", Helvetica, "Hiragino Sans GB", "Microsoft YaHei", Arial, sans-serif;widows: 1;word-spacing: 2px;caret-color: rgb(255, 0, 0);text-align: center;box-sizing: border-box !important;overflow-wrap: break-word !important;"><strong style="max-width: 100%;color: rgb(0, 0, 0);font-family: -apple-system-font, system-ui, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="max-width: 100%;letter-spacing: 0.5px;font-size: 14px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><strong style="max-width: 100%;font-size: 16px;letter-spacing: 0.544px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="max-width: 100%;letter-spacing: 0.5px;box-sizing: border-box !important;overflow-wrap: break-word !important;"></span></strong></span></strong></section><section data-brushtype="text" style="padding-right: 0em;padding-left: 0em;white-space: normal;max-width: 100%;letter-spacing: 0.544px;color: rgb(62, 62, 62);font-family: "Helvetica Neue", Helvetica, "Hiragino Sans GB", "Microsoft YaHei", Arial, sans-serif;widows: 1;word-spacing: 2px;caret-color: rgb(255, 0, 0);text-align: center;box-sizing: border-box !important;overflow-wrap: break-word !important;"><strong style="max-width: 100%;color: rgb(0, 0, 0);font-family: -apple-system-font, system-ui, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;letter-spacing: 0.544px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="max-width: 100%;letter-spacing: 0.5px;font-size: 14px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><strong style="max-width: 100%;font-size: 16px;letter-spacing: 0.544px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="max-width: 100%;letter-spacing: 0.5px;box-sizing: border-box !important;overflow-wrap: break-word !important;">—</span></strong>完<strong style="max-width: 100%;font-size: 16px;letter-spacing: 0.544px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="max-width: 100%;letter-spacing: 0.5px;font-size: 14px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><strong style="max-width: 100%;font-size: 16px;letter-spacing: 0.544px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="max-width: 100%;letter-spacing: 0.5px;box-sizing: border-box !important;overflow-wrap: break-word !important;">—</span></strong></span></strong></span></strong></section><pre style="padding-right: 0em;padding-left: 0em;max-width: 100%;letter-spacing: 0.544px;color: rgb(62, 62, 62);widows: 1;word-spacing: 2px;caret-color: rgb(255, 0, 0);text-align: center;box-sizing: border-box !important;overflow-wrap: break-word !important;"><pre style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;"><section style="max-width: 100%;letter-spacing: 0.544px;white-space: normal;font-family: -apple-system-font, system-ui, "Helvetica Neue", "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei UI", "Microsoft YaHei", Arial, sans-serif;box-sizing: border-box !important;overflow-wrap: break-word !important;"><section powered-by="xiumi.us" style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;"><section style="margin-top: 15px;margin-bottom: 25px;max-width: 100%;opacity: 0.8;box-sizing: border-box !important;overflow-wrap: break-word !important;"><section style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;"><section style="max-width: 100%;letter-spacing: 0.544px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><section powered-by="xiumi.us" style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;"><section style="margin-top: 15px;margin-bottom: 25px;max-width: 100%;opacity: 0.8;box-sizing: border-box !important;overflow-wrap: break-word !important;"><section style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;"><section style="margin-bottom: 15px;padding-right: 0em;padding-left: 0em;max-width: 100%;color: rgb(127, 127, 127);font-size: 12px;font-family: sans-serif;line-height: 25.5938px;letter-spacing: 3px;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="max-width: 100%;color: rgb(0, 0, 0);box-sizing: border-box !important;overflow-wrap: break-word !important;"><strong style="max-width: 100%;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="max-width: 100%;font-size: 16px;font-family: 微软雅黑;caret-color: red;box-sizing: border-box !important;overflow-wrap: break-word !important;">为您推荐</span></strong></span></section><section style="margin-top: 5px;margin-bottom: 5px;padding-right: 0em;padding-left: 0em;max-width: 100%;min-height: 1em;font-family: sans-serif;letter-spacing: 0px;opacity: 0.8;line-height: normal;box-sizing: border-box !important;overflow-wrap: break-word !important;">一文了解深度推荐算法的演进</section><section style="margin-top: 5px;margin-bottom: 5px;padding-right: 0em;padding-left: 0em;max-width: 100%;min-height: 1em;font-family: sans-serif;letter-spacing: 0px;opacity: 0.8;line-height: normal;box-sizing: border-box !important;overflow-wrap: break-word !important;">干货 | 算法工程师超实用技术路线图</section><section style="margin-top: 5px;margin-bottom: 5px;padding-right: 0em;padding-left: 0em;max-width: 100%;min-height: 1em;font-family: sans-serif;letter-spacing: 0px;opacity: 0.8;line-height: normal;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="font-size: 14px;">13个算法工程师必须掌握的PyTorch Tricks</span></section><section style="margin-top: 5px;margin-bottom: 5px;padding-right: 0em;padding-left: 0em;max-width: 100%;min-height: 1em;font-family: sans-serif;letter-spacing: 0px;opacity: 0.8;line-height: normal;box-sizing: border-box !important;overflow-wrap: break-word !important;"><span style="font-size: 14px;">吴恩达上新:生成对抗网络(GAN)专项课程</span><br  /></section><section style="margin-top: 5px;margin-bottom: 5px;padding-right: 0em;padding-left: 0em;max-width: 100%;min-height: 1em;font-family: sans-serif;letter-spacing: 0px;opacity: 0.8;line-height: normal;box-sizing: border-box !important;overflow-wrap: break-word !important;">拿到2021灰飞烟灭算法岗offer的大佬们是啥样的<span style="font-size: 14px;">?</span></section></section></section></section></section></section></section></section></section>

没有什么内存问题,是一行Python代码解决不了的

本篇文章来源于: 深度学习这件小事

本文为原创文章,版权归所有,欢迎分享本文,转载请保留出处!

知行编程网
知行编程网 关注:1    粉丝:1
这个人很懒,什么都没写

发表评论

表情 格式 链接 私密 签到
扫一扫二维码分享